Top
Best
New

Posted by zdw 1 day ago

Creepy Crawlies(people.kernel.org)
1183 points | 594 commentspage 9
vist_orn 11 hours ago|
Real creepy crawlies in the server rack are always a bigger surprise than any code bug.
lxgr 18 hours ago||
> [...] when a source is guaranteed to be LLM-free, like the entire history of kernel commits [...]

Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?

And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)

While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.

And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.

The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).

[1] https://docs.kernel.org/process/coding-assistants.html

alkonaut 18 hours ago||
Proof-of-humanity can’t come soon enough.

We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.

Arubis 17 hours ago|
How do you define humanity?

How do you ensure it includes every human?

How do you ensure it doesn’t include every non-human?

I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.

alkonaut 15 hours ago|||
Every place on earth has some legal definition of who is human. The system I’m thinking of isn’t a technical/captcha one, it’s a human curated list of humans. Just an electronic ID. Those already exist but the challenge is making them (acceptably) privacy-preserving.

I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perhaps over a certain age). And I don’t want the ID issuer to know what site/service asked whether I’m a human or I’m 18 etc.

Kamq 17 hours ago|||
You're not wrong, but something can work well enough to still be useful despite falling short of the idea of a proof or any formal definition.

Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough

Arubis 17 hours ago|||
That’s a pragmatic and understandable argument. And for an individual hobbyist site owner, that’s fine. Are we okay with excluding 1 person in 20 from the services of a midsized organization? What if they’re integral to the workplace? Or a major transport provider without differentiated competitors? What if the organization is a state government?
inigyou 15 hours ago|||
I'm one of the 5% apparently, cloudflare thinks I'm a bot. I suppose you'll exempt me in exchange for all my ID documents and bank statements?
akoboldfrying 3 hours ago||
Are you prepared to randomly serve data that is incorrect -- but in such a way that real people can easily detect it -- some very small fraction of the time?

If so, you could serve iocaine-style bogus pages 1% (say) of the time that:

1. "Look like" real pages to an LLM-less computer (if you get to the point where you have pushed crawlers to use LLMs to detect nonsense, that already increases the cost a lot)

2. Look "obviously wrong" to a human (E.g., you could take some regular text and swap the order of each adjacent pair of words)

3. Are cheap to generate

4. Important: Contain more links than regular pages, on average, and each to an always-bogus page

The idea is that, due to the large number of pages fetched by crawlers, even with a very low "random bogus page rate", like 1%, they will soon unwittingly hit a bogus page, from which point the fraction of their time spent accessing expensive genuine pages will fall exponentially due to the compounding effect of the higher outbound link count on bogus pages. Humans seeing a bogus page will be confused and annoyed, but simply refreshing the page in the browser will solve the problem 99% of the time (and of course the possibility of this happening can be documented, even on the page itself).

The main advantage is that this does not require any IP-based tracking. You could of course decide to apply this only to pages that are already slightly suspicious (e.g., very old commits).

hubraumhugo 6 hours ago||
There is a HN article on abusive AI crawlers on the front page almost every week, but we rarely talk about the path forward. Web scraping has been around for as long as the internet, and it was fine because we had established best practices (rate limiting, self-identification, robots.txt, etc.) that the industry agreed upon. Now we have AI labs and their crawlers that don't care about any of this gentlemen's agreement.

So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?

Some approaches that I think are promising:

- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).

- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.

- what else?

[0] https://datatracker.ietf.org/doc/draft-vaughan-machine-reada...

[1] https://datatracker.ietf.org/doc/html/draft-meunier-http-mes...

Wowfunhappy 10 hours ago||
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that?

And none of them are smart enough to realize they could just `git clone` all the content and use it offline?

micah_chatt 10 hours ago||
If you do the math (also a common system design interview question for an AI lab), its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes. At that price point, its actually very affordable to many thousands of organizations to get their own copy. I would expect the major labs to special case kernel.org similarly to other sites like Wikipedia, but not the majority of scrapers
Wowfunhappy 9 hours ago||
> its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes.

That is super interesting, thank you!

> At that price point, its actually very affordable to many thousands of organizations to get their own copy.

I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?

strix_varius 10 hours ago||
There are many more labs than that, and humans aren't designing unique scraping processes per domain.
sunaookami 5 hours ago||
Have the exact same problem with my MediaWiki and Gitea instances. Doing a managed challenge through Cloudflare (sigh) on "expensive URLs" helped and minimizes the impact on real users. These are also all over a million of residential IPs, very annoying.
singpolyma3 19 hours ago||
Why is no one filing lawsuits over this yet?
Symbiote 19 hours ago||
Against what person or entity?
inigyou 15 hours ago|||
Sometimes the first step of a lawsuit is discovering who you're suing. It's not unusual and there are processes for it. You could bring something like an access log to a court and receive an order for all ISPs involved to unmask the corresponding users.
singpolyma3 16 hours ago||||
Whoever is doing the abuse. If we don't know who that is we should find out
johneth 18 hours ago|||
Bright Data et al.
inigyou 15 hours ago||
Bright Data's business is legal, but they could be subpoenaed to find out which customer is making these requests, but first you would have to prove they were actually involved, because there are many residential proxy providers.
mmooss 18 hours ago||
Indeed. The solution isn't technical but legal. It's clearly abusive of - really stealing - other people's resources; there's no question about it.

For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.

akersten 17 hours ago||
The brazen crime of asking for a resource and receiving it.

You have the power, and should exercise it, to rate limit bad actors

inigyou 15 hours ago|||
How do you know who the bad actors are, if you don't use the legal process to unmask them?
mmooss 12 hours ago|||
> You have the power, and should exercise it, to rate limit bad actors

It's well known, and it was the point of the OP, that we don't have that power. If you know a solution, please share it - the world would love to hear it.

nxndbebdb 20 hours ago||
Just serve the raw commit and render on frontend. I really don't get why they are complaining, just be performant
RussianBot9580 19 hours ago|
And for a shallow clone you would serve... what?
hdbsbs 16 hours ago||
No need for a shallow clone, just let the Frontend fetch the relevant objects from a static file server
Velocifyer 20 hours ago|
But why don't they just git clone?
rcxdude 20 hours ago||
These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking.

(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)

ipdashc 19 hours ago|||
> it's not actually obvious who's running these bots

This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!

Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option

inigyou 15 hours ago|||
Especially since if any crawler IP is in your country, you can get a subpoena to find out who it is.
tmp10423288442 19 hours ago|||
ChatGPT uses a distinct bot type for ChatGPT-initiated web search versus model training:

GPTBot → model improvement / training crawl

OAI-SearchBot → search index / ChatGPT Search

ChatGPT-User → fetch initiated by a user's ChatGPT interaction

OAI-AdsBot → advertising landing-page crawl

Symbiote 19 hours ago||||
Meta is an exception. They intentionally ignore robots.txt rules.

(No second chances, I blocked their whole IP range.)

inigyou 15 hours ago||
Everyone ignores robots.txt because every robots.txt blocks everyone.
nubinetwork 15 hours ago||
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
inigyou 6 hours ago||
Think about incentives, what benefit does Anthropic obtain by going to hell?
voakbasda 19 hours ago|||
Why do we think that only “good guys” are training LLMs? I imagine organized crime is getting in on the game too.
lkbm 20 hours ago|||
Because they're crawling a billion webpages, only a tiny fraction of which can be git cloned, and configuring a special case just for that tiny fraction isn't worth the effort (of the crawlers).
DarmokTanagra 18 hours ago|||
vibe coded crawlers run by morally bankrupt trend chasers aren't going to be the most well engineered systems you come across.
acedTrex 20 hours ago||
Because the crawlers dont care, they are the internets parasites. Their creators care nothing for people or systems downstream of their greed.
More comments...