Posted by sammy0910 6 hours ago
What's the value for the website maker in these? There's user value to it (rather than just value for Anthropic/OpenAI) but the whole point of these bots is to make sure nobody ever needs to visit your website.
1. Use a catch-all subdomain
2. Reject email to addresses where a simple checksum does not match (valid addresses being something like asdfghij.klmn@catchall.domain.tld where the klmn is taken from hash of asdfghij and some secret, anything not matching that pattern gets rejected) [this means running your own mail server so you can do that on incoming mail, running a scan to pick out and remove errant messages regularly, or having a mail client where you can implement this as a plugin]
3. On your contact detail page, don't give out a straight email address but have the the page run some Anubis style JS before asking the server to respond with an address that works
4. Once an address is obviously being used in a manner you don't care to respond to, add it to an explicit block list.
For extra protection from addresses being passed around (intentionally or via hack & other leaks), ask for their email address first and use that as part of the hash, then you can reject mail to your catch-all that doesn't come from the original other party.
Whether all this is worth the effort of implementing, is of course up to you. It probably isn't… I've been considering something like it, but mainly as a fun little project rather than something that would be massively useful.
There you go. Ditch it. Make you pages cachable as much as you can.
The most likely answer is they don't give a crap because it's not their money and no one is going to do anything about it.
That sort of behavior would justify a name and shame imo
Inconsiderate shitty services don’t deserve the politeness of protecting their name
Though they probably use the cheapest possible models they can so the damage from their dumb business model would be limited
Who is?!
You see something, you say something, or you do something.
I experienced the same thing after submitting articles from my small photography website. I did debate using some of the off-the-shelf solutions like Cloudflare, Anubis, or iocaine, but I ended up building my own L7 proxy that utterly fucks the bots up: https://robertmay.photography/journal/bwbach-my-guardian-gob.... I will make it generally available at some point once I get it optimised how I'd like, but currently I'm able to trap about 3000 concurrent crawlers in my tarpit of slow requests on two small Hetzner VMs. And that 3k is really just the limits of the networks hitting my servers right now, I think it'll go significantly higher than that.
I haven't really seen anyone else combat them this way, possibly because many of the standard proxies and servers are set up for fast requests, not serving them as slowly as possible. Deliberately serving page loads for as long as the requester will accept packets turns out to be a really funny way to stop the botnets - when I just rejected the requests they in turn hammered my server even harder. By tanking the request volume with a ridiculous BEAM-based proxy you can reduce their request speed to a crawl and I'm fairly certain a couple of them have blacklisted my server now.
https://deep.43z.one (how deep bots will crawl)
https://stalling.43z.one (how long they'll keep the connection open)
EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!
Looking at you, Meta.
I do still get some requests going up to 15 minutes which is quite ridiculous, though those don't seem to be super common (maybe they're learning).
I think the shortest timeouts I've seen are between 2-5 seconds, but they're surprisingly infrequent and actually that's still a massive inconvenience for them. I did at one point just drop any request I couldn't keep open for more than 5 seconds but it actually caused more server load. I'm essentially inverting the server load by making slow requests more efficient, which is counter to everything I ever learnt about performance for web serving :D
I think the average I see is probably somewhere between 15-30 seconds. I'm guessing many crawlers don't want to go too low because they'll be trying to fetch large files like images, or perhaps they know they'll be slowing servers down and want to ensure they get the response. Most likely, the majority of crawler authors aren't expecting some arsehole to exploit this and they've just left the setting as the default.
I gave up on tarpits when it got to 130k "waiting" connections.