There's something missing from the picture. The bots are:

- Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient.

- Putting in tons of CPU cycles to surpass the Anubis PoC.

- Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions.

This sounds more like escalating DDoS than AI scraping.

I wonder if they could pre-render the stuff older than a month ago and compress it and serve it as static content. Not optimal, trades space for CPU, but might be cheaper.

I also had this problem, but since nobody actually uses my gitea site besides crawlers, I just let a script run through my access log and ban every IP address who asked for a commit in the last 24 hours

Why can't they just ask Linux Foundation for 96, or even 1696 cores?

If you're reading this - go ahead and see HOW Linux Foundation spends their money.

I wonder if we're back to peer to peer networks with proof of useful work (e.g. serving read requests) vs proof of wasted work.

Great chart! Does anyone know what tool was used to make this?

There is a HN article on abusive AI crawlers on the front page almost every week, but we rarely talk about the path forward. Web scraping has been around for as long as the internet, and it was fine because we had established best practices (rate limiting, self-identification, robots.txt, etc.) that the industry agreed upon. Now we have AI labs and their crawlers that don't care about any of this gentlemen's agreement.

So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?

Some approaches that I think are promising:

- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).

- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.

- what else?

[0] https://datatracker.ietf.org/doc/draft-vaughan-machine-reada...

[1] https://datatracker.ietf.org/doc/html/draft-meunier-http-mes...

Reminds me of debugging production issues at 3 AM. Both can make you jump out of your skin.

Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.