Posted by sammy0910 7 hours ago
https://pagespeed.web.dev/analysis/https-sprinklz-io/bmc4mv1...
I experienced the same thing after submitting articles from my small photography website. I did debate using some of the off-the-shelf solutions like Cloudflare, Anubis, or iocaine, but I ended up building my own L7 proxy that utterly fucks the bots up: https://robertmay.photography/journal/bwbach-my-guardian-gob.... I will make it generally available at some point once I get it optimised how I'd like, but currently I'm able to trap about 3000 concurrent crawlers in my tarpit of slow requests on two small Hetzner VMs. And that 3k is really just the limits of the networks hitting my servers right now, I think it'll go significantly higher than that.
I haven't really seen anyone else combat them this way, possibly because many of the standard proxies and servers are set up for fast requests, not serving them as slowly as possible. Deliberately serving page loads for as long as the requester will accept packets turns out to be a really funny way to stop the botnets - when I just rejected the requests they in turn hammered my server even harder. By tanking the request volume with a ridiculous BEAM-based proxy you can reduce their request speed to a crawl and I'm fairly certain a couple of them have blacklisted my server now.
https://deep.43z.one (how deep bots will crawl)
https://stalling.43z.one (how long they'll keep the connection open)
EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!
Looking at you, Meta.
I do still get some requests going up to 15 minutes which is quite ridiculous, though those don't seem to be super common (maybe they're learning).
I think the shortest timeouts I've seen are between 2-5 seconds, but they're surprisingly infrequent and actually that's still a massive inconvenience for them. I did at one point just drop any request I couldn't keep open for more than 5 seconds but it actually caused more server load. I'm essentially inverting the server load by making slow requests more efficient, which is counter to everything I ever learnt about performance for web serving :D
I think the average I see is probably somewhere between 15-30 seconds. I'm guessing many crawlers don't want to go too low because they'll be trying to fetch large files like images, or perhaps they know they'll be slowing servers down and want to ensure they get the response. Most likely, the majority of crawler authors aren't expecting some arsehole to exploit this and they've just left the setting as the default.
I gave up on tarpits when it got to 130k "waiting" connections.
At first, I assumed this was a single blog post, especially with a "Table of Contents" at the top. I guess it's following some old-school blogging conventions, but AFAICR, even those sites retained individual pages for separate posts.
Just merely existing will get you scanned these days.
One possible way around this is to use a wildcard and not serve anything on the base domain. Unless the right subdomain is listed somewhere, they'll not get any scans. Maybe useful for projects that you have online for just yourself and a small closed circle.
>So I took a look at some data. I added top-level logging for every endpoint (how many times was a request made to each url path), and I found that a majority of the requests did not even make an attempt to fetch news articles! For reference, my website is built as a single-page-application, and what this means is that a majority of the requests did not even run the javascript necessary to render the page (!).
So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
>many, many requests per minute?
How many was it?
Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
But hey. Silver lining: free load testing! ;)
i dont think this was the cause of the slowdown, it was just sort of the thing that led to the discovery that this was mostly bots.
in general, because the purpose of the website is to allow users to generate news feeds based on custom criteria, it does do a bit of meaningful work on each request (ones that do follow through and load the full page).
> How many was it? > Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
I'm intentionally leaving some of the details vague here, both because I don't want to be too harsh on the original author (it may have just been accidental/a buggy program/etc.), and because I don't want the traffic generators to know how im flagging them.