Posted by blakesterz 3 hours ago
I just make sure all my pages are super lightweight to cache and serve from cloudflare and let the bots do what they like.
If I put something on the internet for all to see, I want bots to be able to see it too.
I've been on the other side of bot blocks - when I ask my LLM to buy me X and it comes back and says the only place it could buy from is some Bulgarian seller because everywhere else has anti-bot scripts, I make sure to give that Bulgarian site my business.
But there are bad actors - bots that try every possible urls and traversals and cross scripting and so on… I see it in our access logs. This cannot be solved otherwise than some greylisting/blacklisting on ISP level. E.g. this is so huge that we have started to block all Chinese inbound traffic. I am sorry for that because Deepseek and other good actors are probably blocked as well but IP rotation made it impossible to filter bad ones only… (and I suppose that DS uses EU or US addresses as well so they shouldn’t be blocked in full).
Indeed, they use an edge SSL Bump proxy that is already bandwidth limited and has to speak real HTTPS.
I do not focus too much on IP addresses but rather the behavior that they can not hide. LLM's can help you spot this behavior. Good LLM vs. Bad LLM. Gemini knows a few tricks but you have to coax it out of them. It seems to not want to overload people with information. That said there are some good IP lists that can be used in blackhole routing tables, nginx geo tables, etc...
I focus on content on our site so blocking part of internet with wordfence was brute but as life has only few thousands of weeks, it was the least I wanted to think about :).
Would you mind to share a source of good IP lists pls?
LLM's can help you implement iprange into IP list building scripts to keep the lists smaller and more efficient.
[1] - https://github.com/firehol/blocklist-ipsets/
I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.
The bots may have layer 7 signatures that make them "easy" to block, but I'm dealing with COTS software with no ability to make changes to the stack. Even if I could by the time they're blocked the work is done and they're never coming back anyway.
I don't think I'm seeing the LLM bots in the scraping on this particular site. This site's data is highly sought-after by data brokers, legal services advertising, debt collectors, etc. I tend to think I'm seeing homespun scrapers. This site has been up for 20+ years and the scrapers seem to be well-adapted to using the HTML as an ad-hoc API for the data. The difference in the last few years has been the availability of vast numbers of residential proxies to mask the traffic.
Some of the things that seemed like they'd cut out lazier bots/scripters involved browser fingerprinting that I didn't want to yet have to deal with disclosures or approval for. But also even some of these browser fingerprinting things were disable-able via flags when you start up Chrome or whatever.
I think what would work overall is to track all requests for a pattern of page access and start blocking other systems that are part of that pattern. But that also wouldn't account for like... landing on the front page of HN, which happens not infrequently.
All of this is aside from per-IP or per-user proof of work. Whenever someone lands on a more expensive page I make the browser do proof of work and store it in a cookie, unique to the IP or user. (Signed in users don't have to do proof of work). This is a perfect solution except for the IP rotation I'm talking about above.
edit: ah also, I don't do SSL termination so can't spot the SSL configuration. But Cloudflare does. And all of this is even after Cloudflare paid bot traffic systems are in place.
edit 2:
> tiny percentage of bots will still get through but the load becomes much more manageable.
The load is not the problem. It's only at most a few bot hits per second. I'm just having trouble removing bot attribution from view counts.
Youtube wont count a view until you watch for some percentage of the video. Maybe consider something similar.
I love sharing methods with the botters so they step up their game, thus forcing me to step up my game. Everything I use does not require javascript, cookies or authentication. I save authentication as a last resort which means I lost that round.
Botters are welcome and encouraged to test their setup on my silly hobby sites. Here's one of them: [1]
[1] - https://altchan.org/sh/33
P.S. These are the simplest crude methods for the obvious bots, let your LLM to help spot the fakers. pre-2026 Safari will not work. Do not visit image boards with unpatched browsers.
i heard hcaptcha is good
The problem with the libraries is that all these scans are behind walls, so while the texts are public domain nobody can easily access them. I'm trying to liberate them and re-upload them outside the walls.
Until recently a lot of libraries would allow you to remotely sign up for a library card, but in the last 12 months almost all of them have reverted to having to bring ID in person to the library to validate your account, which puts the brakes on things.
- Residential proxy providers
- Hosting providers with IP-rotation as a service.
- Hosting providers who chase AI training customers, DigitalOcean is a big one
- Clouds like AWS with headless browsing as a service
- Platforms like Replit whose value proposition starts to fall apart if clients can't make requests from their platform.
- Companies like meta, whose existing bots are now obviously completely unchained from their advertised responsibilities
This is the way for companies these days to avoid legal responsibility for their actions - hire a contractor on a "don't ask, don't tell" basis.
Of course such things are extra-ironic in the context of US companies complaining about having their condensed by Chinese companies.
Edit: I read this in one article a while back. I can't links either corroborating or refuting it now and I think that's again because whatever the companies do, they want to keep it hush-hush.
https://git.eeqj.de/sneak/smallwebwaf
It’s alpha but it should be ready for real-world testing in the next couple of days. I’m going to drop it in front of my Gitea and home-grown api servers.
I’m super tired of Anubis spamming me with furry fetish art on its pointless interstitials 20 times a day. As someone pointed out in another comment, LLM scraper bots are super easy to detect with basic heuristics. PoW and session cookies and an interstitial are massive overkill and annoying spam.
[1]https://www.makonea.com/en-US/casual/scraping-bots-are-destr...
RIGHT?
"Open Access" means supporting users. Now your users are also bots, and the people that deploy those bots. You need to be able to serve those users too.