Posted by xena 14 hours ago
Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.
The situation you are in is far less dire sounding when you consider that you have these options available to you.
It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.
Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.
- https://techaro.lol/robots.txt
- https://anubis.techaro.lol/robots.txt
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
I've just finished creating a WASM 1.0 implementation (simple JIT on x86 and x86_64) for plugin usage and I'm looking on how to support more languages beyond C (for which I have "micro-libc" to create really small binaries). In particular C++ and Rust so the article contains useful info for that. C++ that would be more problematic I think, would have to resort to full Emscripten or something... I've tried to compile standalone C++ stdlib but wasn't much sucessful yet.
BTW, I see in the screenshots you also test Firefox, I think it would make sense to test the latest two ESR versions (these are often used by Linux distributions).
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
Yes, but you normally don't try to fight all strategies up front, you add stuff on a need-to basis. Currently, lots of people are in the stage of "need to get this thing using bunch of IPs to stop hammering my servers", apparently Anubis helps them with that. If that stops working, other tactics are tried, and so it goes on for the life of your service. Hasn't stopped yet, doesn't seem to be stopped soon either.
Turns out that in practice there's some bots that implement support for this but not JavaScript.
People have been sent to jail for participating in LOIC attacks which were less destructive that what these bots do on the daily.
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
"Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.
And then regular users: who now can't really browse your site.
And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down.
The solution basically makes the favored customer well funded AI scrapers, not users.
Extreme configurations with high difficulty do make for a frustrating experience on very old devices, but in practice it rarely gets in the way.
Only the tail end of what one could consider normal users (those running custom browsers or having JS disabled or nonfunctional) run into issues here, but they'll have these issues with any other bot-blocking mechanism that still offers some protection these days.
Apple's "ask your vendor for tokens to exchange for access" web standard gets around this, assuming those bespoke and Javascript-less browsers implement PATs, but that standard seems to be completely dead in the water outside of Safari.
That's entirely up to how the site operator configures the PoW challenge.
> anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down
Do you have any evidence that it is the large well funded AI companies DoSing sites by (for example) scraping absurd numbers of permutations of cgit url parameters?
Even if it is (which I doubt), if all this accomplishes is getting them to spend 10 minutes optimizing their bots to not do that it will be a major win.
https://github.com/eternal-flame-AD/pow-buster - Browser extension that already used WASM to accelerate the anubis solver, among others.
https://github.com/kasper93/anubis_webgpu - Browser extension that uses webgpu, for another order-of-magnitude speedup (depending on your GPU of course).
That would defeat the purpose. The goal is to make scraping costly, not profitable.
But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.
Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.
This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.
> This makes Anubis challenges use a memory-hard proof of work function (argon2id) instead of just a CPU hard one. It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
Nice.
Meanwhile BOINC is well-established as the platform for distributed computation. If you can figure out how to squeeze its work units into Wasm challenges, I'm sure a lot of researchers would thank you.