Posted by chmaynard 20 hours ago
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
it could be RAM-bound, which is very much NOT cheap nowadays :)
100MB for 1 second just is not much of a deterrent.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.
I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.
reason why is 1. Google and others really needed the training data, and 2. it probably helped justify the cost of providing the captcha service for free worldwide (old free tier was 1M/mo)
I don't think you'll find an article by Google saying "yes, we sometimes completely block users while making it look like they're not blocked and wasting their time".
Google also prefers if you have a Google account logged in.
funny with all the IP information they have, cloudflare cannot do a better job. (I am on IPv6)
and most of the time, its on marketing product pages like in framework main site, which can be cached.
If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing for.
I dont care what recaptcha wants to optimize for. I dont think that using a vpn is that a rare thing anyway. If others have figured out how to do it without requiring spending 30 seconds to solve a captcha, I dont see why websites still use recaptcha/captchas for that.
And that it is "my fault" not being logged into google I was least expecting to see here.
Parent was just starting a fact of how our digital overlords determine the probability of your browser being a bot. Why take it personal?
https://www.kaspersky.com/blog/what-is-wrong-with-free-vpn-s...
Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult challenge) in the exact same scenario.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
Reload loops, or being able to "bypass" anubis (unless you merely mean bypassing it for the token validity period by solving a challenge), sound like misconfigurations. There's no reason for anubis itself to cause reload loops; it's tricky to configure a webserver to use it in some scenarios.
Any ability to bypass anubis probably means the site is using it in auth/challenge mode only, and then misconfigured their webserver's auth checking. Or it's a bug. If you mean the double-spend tavis mentioned in his blog post which previously made the HN frontpage, that was patched right after it was reported to the maintainer almost a year ago.
Do you have any evidence that AI providers aren't using residential proxies?
If I hear the fan spinning at night, you're probably getting caught immediately.
If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV playback will start to stutter, which will be resolved by taking the compromised TV box, and removing the malware using advanced mechanical means called "a trash compactor".
video decoding is hardware accelerated, and there's probably enough excess compute to be able to do some sort of PoW challenge. Besides, unlike humans, bots aren't in a hurry, so they can spread out the work across a long time to minimize disruption.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).
They meant you can’t PoW every page transition.
If clicking every link on your website throws you back to another Anubis page for 2-3 seconds, users will bounce.
That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.
We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.
Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.
These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.
This isn't to say that true botnets built with actual malware delivered through either software exploits, phishing attacks, or watering hole attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
I believe that's the HN application itself, not a WAF in front of it.
Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.
no options to delete content beyond a narrow window.
Poor accessibility
bad mobile support
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.
Using VPN? Blocked.
Your iPhone is too old, blocked.
Your screen brightness too low? Believe or not, blocked.
I've ended up putting only IPv6 on the domain. It's running this way for 2 years already.
... What?!
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
Private trackers do this. If they ban a user that geolocates to a certain city and ISP, they'll ban new signups from that city and ISP because there's probably only a few users from the same city and ISP. And then report to their friends at other trackers, that a user with that city and ISP is trying to evade a ban.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
I can imagine that sites with dynamic content and potentially unbounded query types or pathnames are in danger from particularly stupid crawlers.
Grok actually shows a number of sources used for an answer. Once I asked it something simple and it apparently scanned 200 different websites. And it was just a short prompt. Now imagine millions of users asking for something multiple times a day.
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
Who’s doing it, are they even using the data?
Millions per hour is tens per second though; perhaps the fix is performance improvements
That'll be great until.. they rewrite the scrapers in Rust! Then we're really hosed!
Maybe every web query for Linux commands in $LARGE_COUNTRY checks all the Linux websites again.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
Which powerful entities have historically hated a free and open internet?
...all of them??
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access.
But really, this finally explains the hordes of really basic boring games that just advertise other boring games. Idle games and the like that really just want you to keep your phone unlocked and open. Millions of downloads on the app stores for entirely offline content (and ads) and no way to block the network access.
These aren’t as simple as downloading a free game and then the phone is compromised as long as it’s installed.
The users who install these things don’t care about permissions prompts. They’ll follow instructions to tap any prompt the instructions ask. They want the free thing and don’t care what they have to do to get it.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
If only you were the only one doing it...
I wrote about this recently as well.
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
Grub was, in a very real way, a botnet. And, we harmed site owners when we were operating at full capacity. There were a few bugs in the early days where we would reschedule a site because the ingestion in the server broke, which then caused the page to be rescheduled. Stupid error, and we fixed it, but it's illustrative of the fact even good intentions isn't enough here.
What I've come up with over the years is similar to the idea Cloudflare is implementing with payments to site owners by charging the crawlers. My objection to Cloudflare's implementation is based on a personal opinion about Cloudflare being a single point of failure and also a decrypted choke point. Their ideas about how to handle crawlers, and pay for the load on the sites is solid. It presumes to use the 402 response to demand payment. I'm clearly biased about Cloudflare, but that's my prerogative here.
It may be possible to solve this with cryptocurrency, in a distributed way, and I've prototyped a system that uses the Lightning Network to handle the payments from a 402 response. Lightning Labs also worked on a project called Apeture for a time that did something similar.
HN's site knows every item ID, and it knows fresh IDs get read in a predictable distribution while old ones mostly sleep. Sustained access outside that is itself the scraper signal. No IP reputation needed, which matters now that residential proxies burn an address after a handful of requests.
Karma gives you a clean way to let humans through. Issue logged-in accounts with decent karma a token whose cold-content budget scales with it (the karma), so an account with history scrolling back through a 2014 thread just reads it. Karma should gate the tier, not be spent as currency, or upvote rings become a crawling business.
Anonymous readers who deep link into one old thread from a search engine get the first fetch or two free (and you watch the article IDs, not the IPs). What remains after those carve-outs is bulk traversal of cold IDs with no identity attached, and that traffic gets rate limited and answered with a 402: pay per page over Lightning, priced at a healthy multiple of what residential proxy bandwidth already costs, or come back slowly for free.
There are probably holes in these thoughts. It's one of the harder problems to solve, for sure.
On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and therefore there is no reason for the existence of that traffic (at least as far as I'm concerned), and therefore I treat it as malicious.
The address is recorded and goes into a database.
Periodically, the database is dumped to a file in a format that the firewall reads, and all the 'malicious' addresses detected above are added to a list so that those addresses are blocked from accessing the legitimate service ports. (analogy: if you throw an egg at my outdoor wall, I'm not going to let you into my house through the door because I don't want egg on my furniture).
I have a blocking period of about 3 months - because the things I run are important to exactly a single person. A blocking period much shorter would be recommended to prevent the gross-overblocking of legitimate users who may have un-lucked into being assigned a residential IP address that was previously used in a proxy-scan-scam.
Discard this if it's a stupid idea at-scale, but I quite the like the 'idea' of it, and I made it work, mainly for the technical challenge.
Project is here on Github: https://github.com/UninvitedActivity/UninvitedActivity
Is that a side effect of whatever you are doing?
There are other reasons why we might not respond, but overload is the main one.
Edit: I only see one email in the archive related to your account. It was from Sept 2024 and we responded to it. Are you talking about a different account?
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
<https://news.ycombinator.com/item?id=32048148>
<https://news.ycombinator.com/item?id=32031243>
(AFAIK that specific failure mode has in fact been addressed.)
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
<https://news.ycombinator.com/item?id=36078578>
<https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard.
API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites.
That said, a standard, say, JSON packaging of website contents available on request might be fun to have.
You're right to point out that if you're trying to get the contents of dead objects the API is of no use though.
On dead/flagged items, there's some value. Whilst the title/URL context aren't available, just knowing what fraction of submissions and comments are moderated is interesting data, and it is possible to construct patterns against specific accounts.
I'm frequently encountering what appear to be banned accounts. Being able to trace those through the API to see where and when they were banned, or now much moderated activity they're generating, can be useful. I'm relying heavily on the "/replies?id=<UID>&by=<moderator>" search endpoint (generally dang, tomhow, sctb, or pg as mod) currently to find out if there was a specific ban admonishment from a moderator. That's often but not always the case.
But it's not possible, say, to tell through the API what sites are banned. Looking at site history with "showdead" enabled can tell you that though, e.g.:
<https://news.ycombinator.com/from?site=synthetica.cloud>
(From the New queue, one of several "dead" submissions not flagged, suggesting a site ban.)
Hypothesizing an undocumented "site" API endpoint ... doesn't seem to check out:
https://hacker-news.firebaseio.com/v0/site/synthetica.cloud?print=pretty
not found
And there's the "day" endpoint doesn't seem to work either, though it conspicuously does not report "not found", e.g.:
https://hacker-news.firebaseio.com/v0/day/2025-07-10?print=pretty
... it's starting to make sense, but ...
... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:
<https://news.ycombinator.com/front?day=2008-11-05>
It's the collection of 30 items from that date I'm interested in. For my scraping, I don't actually need to further query the individual posts as I've got the elements I'm interested in (title, date, story position, URL, votes, comment count, submitter, site/domain) from the index page itself, parsed out of the HTML. The "Past" entries alone are a significant (though not huge) request load. To update the past three years would be about another 1,000 requests, which, if fulfilled and modestly rate-limited would hopefully not keel the servers over.
Once I've pulled in those "Past" pages, I could of course do further API queries, though at this point I don't see any specific need to do so.
I suppose that requesting the "past" links be included in the API set could be a request I might make of HN, or the ability to request, say, all submissions (or comments!) for a given date.
There are groups which have done HN analytics in the past using the API, for example Whaly.io:
"A Year on Hacker News" <https://news.ycombinator.com/item?id=31295219>
"Top Hacker News commenters of 2021" <https://news.ycombinator.com/item?id=29778994>
"What Happened This Year on Hacker News (2021)" <https://news.ycombinator.com/item?id=29769470>
I could look more into their methodology to see if I can use similar approaches.
The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)
OK, looking at a recent dead atomic128 comment:
$ curl -s 'https://hacker-news.firebaseio.com/v0/item/48820709.json?print=pretty'
{
"by" : "atomic128",
"dead" : true,
"id" : 48820709,
"parent" : 48819517,
"text" : "[flagged]",
"time" : 1783444517,
"type" : "comment"
}
And from a current dead submission in the New queue:
$ curl -s 'https://hacker-news.firebaseio.com/v0/item/48868688.json?print=pretty'
{
"by" : "millwright-sw",
"dead" : true,
"id" : 48868688,
"score" : 1,
"time" : 1783743361,
"type" : "story"
}
To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...
It's important to note that neither side has moral legitimacy. Not everyone who carries a rifle is a enemy. Not everyone wearing body armor is a saint.
I have given up on the idea that "human vs bot" matters at all when it comes to anything other than voting (which should only be done in person with paper and pen, by the way.)
You could make an argument that "likes" are a form of voting, but you shouldn't. We need to abandon the idea of supposedly democratized algorithms and focus instead on actual democracy.
On my sites, I see ClaudeBot consistently (and has been like this for over a year) ask for "${SITE}.com/base_dir1" and then get the redirect (Caddy does this automatically) to get "${SITE}.com/base1/base_dir1/" (trailing slash).
The hrefs on my sites include the trailing slashes for directories, so looks like ClaudeBot's internal code is stripping them off before requesting them, and therefore essentially makes almost 2x the requests to my sites for directories, half of them ending up being redirects back to the same url but with the trailing slash.
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.
The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.
I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.
Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.
Instead, exchange web traffic for actual $. Say, some kind of tokens that are easily turned back into hard cash through a 3rd party.
Requesting a 100KB file? Okay, that'll be a $0.00002 token, please! (visitor's user agent provides it in a manner transparent to regular web users). Requesting a 3MB image? Okay, that'll be a $0.0005 token, please!
Result: niche websites earn hard cash. It doesn't matter much if you're hammered as long as the hammering comes with a corresponding flow of tokens (read: $). No token(s)? No service.
Regular web users would pay for those tokens through their normal internet service fees, and otherwise not be bothered. Massive scrapers would have to pay somehow for the tokens to be served web data at all.
In effect: put the bulk of public web sites behind a paywall. But with the bar low enough & in a manner that it's transparent for regular web users. Clicked "reload" by accident? Oops, internet service bill got upped by 2 micro-$.
Also all major browsers block crypto miners on webpages now (for good reasons) so it may prove difficult to allow "good" mining scripts while still blocking "bad" ones.
I don't think this is a practical solution
I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The PoW challenge prevents less than 10% of the bots.
As long as the website gets paid more than the cost of serving the pages, it does not matter if a human or a bot did the POW.
Securing signup forms is another issue. Maybe related. But not what I was referring to.
> partly because it causes annoying delays for those trying to get to the site
This is true but usually a small issue. It’s further alleviated by cached tokens so you only have to solve the challenge once in a while per site, and a login token may let you skip it.
> partly because it seems inevitable that the scrapers will eventually find their way around it…A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
Solved by making money off it.