Except the majority of LLM training content nowadays is synthetically generated by LLMs. I wish people would stop making this statement, I don't know why this claim persists to this day. It wasn't true two years ago and it sure isn't true now
And that's just known AI, I can't imagine what requests are pretending to a real person when they aren't.
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.
Some ISPs ban customers based on a single report there - have fun!
You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage".
If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally unaware and non-consenting to residential proxy use.
You are not, in fact, hurting the residential proxy-ers at all. Not one bit. They will move on to another IP and another compromised LAN, and they will continue to move on and on and on. They will not be harmed or impeded; they will simply keep turning up fresh, new, high-reputation IPv4 and IPv6 sources. This is a sheer numbers game, where the numbers are always in favor of the attackers.
Also if network admins keep blocking/filtering abusive residential proxies, they will balloon their firewall rules and cause actual performance issues at the network level. You will turn into your own DDOS without any actual benefit. You're on the losing side of the numbers game, and in the immortal words of W.O.P.R., "The Only Winning Move Is... Not to Play."
How many times do I have to hurt them before they decide to buy a different smart TV?
Seriously, that's like saying "if you try to stop your neighborhood rodent problem by getting citations sent to people with cat food on their porch, you're just hurting the innocent outdoor cat owners". They're participating, whether they know it or not. We can and should PSA and shame and regulate away residential proxies on the supplier side, but we can and should also simultaneously discourage them on the end-user side as well.
And similar to how most people running mail daemons are using blackhole lists nowadays and are keen to not end up on there, maybe ISPs and web hosters can use the AbuseIPDB to sort out their customers.
Just doing nothing doesn't appear to stop the scans hammering my poor Raspberry Pi serving my few Git repositories.
Now in 2026, running a "public proxy" doesn't take an administrator. You don't even need to be aware. Most victims are unknowing victims. They simply subscribe to an ISP and they have their own devices. They are being exploited for that innocence and ignorance. Most victims have no visibility to even detect that they're being used as a proxy. Most victims couldn't stop it, even if they wanted to.
I challenge anyone with a home router to list the processes running on that router, and list all current open connections on that router, and list all open, listening sockets on that router. I bet you can't do it. There are no consumer router OS that lend themselves to being secured, or even diagnosed. Malware can easily be planted on any of them and run, completely invisibly.
A residential proxy server could run on routers, could run on a switch, could run on your "Smart TV" or a smartphone, or a notebook computer. It could be anywhere in any form. Perhaps you consented to it, perhaps you didn't notice.
In no way is this the same as an SMTP open relay situation. If you wanna play "whack-a-mole" with a "blackhole list" you're simply going to overwhelm those lists with false positives and collateral damage. The residential proxies have long since moved on. You won't even find the culprits using those addresses you just blocked. You're just clogging up your own machines. It's a total self-own.
Also, many plotting libraries include an xkcd style these days:
https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot....
And libraries for various languages:
https://github.com/timqian/chart.xkcd
So if you have a preferred dev environment there's probably a way to set it to xkcd style.
So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?
Some approaches that I think are promising:
- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).
- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.
- what else?
[0] https://datatracker.ietf.org/doc/draft-vaughan-machine-reada...
[1] https://datatracker.ietf.org/doc/html/draft-meunier-http-mes...