I've seen this too.

I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..

They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.

Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.

AI has simultaneously made the easiest parts of web development even easier while making the hardest parts near impossible.

Isn't it perfectly reasonable to require an account for any use, and to ensure that making one has a high level difficulty anubis challenge or delay ?

I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging.

That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.

There is a HN article on abusive AI crawlers on the front page almost every week, but we rarely talk about the path forward. Web scraping has been around for as long as the internet, and it was fine because we had established best practices (rate limiting, self-identification, robots.txt, etc.) that the industry agreed upon. Now we have AI labs and their crawlers that don't care about any of this gentlemen's agreement.

So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?

Some approaches that I think are promising:

- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).

- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.

- what else?

[0] https://datatracker.ietf.org/doc/draft-vaughan-machine-reada...

[1] https://datatracker.ietf.org/doc/html/draft-meunier-http-mes...

There's something missing from the picture. The bots are:

- Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient.

- Putting in tons of CPU cycles to surpass the Anubis PoC.

- Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions.

This sounds more like escalating DDoS than AI scraping.

WordPress powered a huge number of websites. Yet the crawlers all go straight for the HTML of those sites rather than the more efficient and structured JSON API which all WordPress sites have.

If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?

Have the exact same problem with my MediaWiki and Gitea instances. Doing a managed challenge through Cloudflare (sigh) on "expensive URLs" helped and minimizes the impact on real users. These are also all over a million of residential IPs, very annoying.

I wonder if they could pre-render the stuff older than a month ago and compress it and serve it as static content. Not optimal, trades space for CPU, but might be cheaper.