Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
The following from Claude: """ A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average: 8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes """
> Us PHP developers
I can tell.
Imagine if WooCommerce had a route "/product/<sku1>/compare/<sku2>" which displayed an auto-generated comparison between any two product pages.
Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text.
Now imagine scrapers are permuting those URLs.
How would you cache that?
That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.
Is it foolish of me to have expected more from a blog post on kernel.org?
The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.
If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.
I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?
It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).
Things just don't add up.
But if bots a crawling diffs between all possible commits it's crazy.
CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.
“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".