Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.
I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.
There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.
If you know CloudFlare as anything else, it speaks to how successfully it has grown and marketed itself into other areas.
I must be missing something. How can using peoples' TVs as bot farms be even remotely legal? Especially when the purpose is to avoid IP blocks?
For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data.
But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots".
I don't buy the explanation that there are millions LLM that are constantly trained on redownloaded data from kernel.org. What would be my better guess is that it is not training, but users are actually accessing this content through chatbot and co. Like when you ask why your sound is suddenly not working anymore after an update or why your wifi driver is constantly disconnected after leaving sleep, it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
But actually, regarding this article, I'm quite amazed that with all the advances of the linux kernel, and server softwares, and that the C10k challenge is solved since a long time, still such a basic traffic is such an issue.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. 14 cpu looks nothing to me. It's like you have 1 iphone and 1 raspberry pi active in a corner of a room. Counting in "seconds" of activities, easily shows meaningless huge numbers. Do you want to know how many breaths I take per year? 8 to 9 millions!
Most certainly, the usage of this shitty Anubis has ruined the climate million times more only with the wasted cpu resources of legit users...
But moreover, by definition the git commits are not supposed to change, ever, so can someone explain to me why the fuck do kernel.org "re-render" the commit to html each time someone is accessing it instead of using a cache or a static version of the html of this commit?
> oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits Again, reading that, my immediate thinking is that it is a shame that such talented people would not be able to have a proper optimization, so that getting the 922 duplicates are just costing a fraction millisecond more after the first person retrieve the first page.
This has nothing to do with that.
Any Node.JS application will happily accept 100K connections. They'll all wait for the under-resourced database behind it. That application "solved" the C10K challenge, but it's still overwhelmed.
> it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
The article describes random algorithmically-generated traffic arriving in batched waves from laundered residential proxy IP addresses, a few unrelated hits in a group then gone. That's not the pattern you'd see if end users were asking their agents for help.
> it is a shame that such talented people would not be able to have a proper optimization
It's mostly not static content in the sense that you're implying.
Routes that access a single commit can be cached. But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
I'm sure they're already caching their useful-to-real-humans data. As the article said, the vast majority of their traffic is bots hitting those arbitrary, permuted URLs. So whatever cache they're using is probably a) missed almost every time, and b) constantly getting evicted to make room for data served to bots (unless they eschew caching to avoid this--fair--and are thus back to the original issue regardless).
There is no "proper optimization" here. It's not slow to go compute the diff between a random pair of refs, render that into pretty HTML, and serve it. But it costs something more than a cache hit, and doing that dozens-to-hundreds of times a second constantly consumes resources.
> But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
Looks like to be wrong based on the article: Today, git.kernel.org receives about 6M daily requests demanding to see random commits.
Also, in "how bad is it", there is a bar graph of the tasks requested. And only diff can't be cached, and it is clearly not the most important one.
> batched waves from laundered residentail proxy IP addresses
Again, I don't see anywhere anything stating that there was "grouped" requests that are directly related, except in something not related to AI: The thing that usually takes us down are not scraper bots, but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. (Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.)
Just "waves" of requests coming from the same "residential proxy IP". But that makes sense because they might come from a same service without being a single "source" trying to scrap for training.
Let's say that I'm chatgpt, or Claude, and I have a lot of user related requests to do things based on the commits there, each one will be a different task, but they will all go through the same pipe, that would be a proxy with the residential IP if the provider notice major websites like this one blocking requests for bots.