The internet is big, but it isn’t that big. I’d expect to see a sudden dropoff as they start re-checking content that hasn’t changed, with some sort of exponential backoff.

Instead, my takeaway is that they are AI crawlers aren’t indexing to store in a way we’re used to with typical search engines, and unilaterally blocking these crawlers across the board would result in quite the “effect”.

Web Bot Auth

https://news.ycombinator.com/item?id=45055452

So interesting they are orders of magnitude worse than the others with the crawl:user-request ratio... noted