Top
Best
New

Posted by chmaynard 22 hours ago

An update on residential proxies and the scraper situation(lwn.net)
274 points | 291 commentspage 2
Bratmon 20 hours ago|
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
inigyou 6 hours ago||
And it's made necessary because another group of people thought that selling IP blocking services would be a good idea. One party sells walls, another party sells ladders.

Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.

TurdF3rguson 18 hours ago|||
I think they also have to operate in countries that don't mind shady things like this.
solidasparagus 14 hours ago|||
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
thomasahle 19 hours ago|||
TIL:

> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.

WTF. That's just botnets.

Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...

jolmg 18 hours ago||
Mmm... your quote (IDK where it's from) mentions them having consent from device owners, but your FBI link cautions on how to avoid getting infected by malware.

If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.

With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 address for all their employees, and most VPN services.

not-a-llm 17 hours ago||
by consent they mean a dialog/EULA with careful wording was display and the user clicked ok.
Barbing 14 hours ago||
And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question.

There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?

jolmg 13 hours ago|||
> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP.

Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.

inigyou 6 hours ago|||
There are absolutely no actual drawbacks for most users.
BoorishBears 19 hours ago||
Thank god for residential proxies.

Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet

Bratmon 18 hours ago|||
By providing a way for corporate AI scrapers to operate with impunity and force the last few independently-run websites to move to the cloud?
inigyou 6 hours ago|||
By providing a way for independent hackers to extract data from closed commercial websites.
BoorishBears 17 hours ago|||
No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies

They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.

Bratmon 15 hours ago|||
I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.

The article was called "The one we're commenting on"

BoorishBears 15 hours ago||
> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.

Does that sound like your typical self-hosted blog?

simoncion 11 minutes ago||
> Does that sound like your typical self-hosted blog?

When compared to

> They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.

yes, absolutely.

gamesieve 7 hours ago|||
The Bright Data mentioned in the article, as well as other similarly malicious but even harder to identify parties, most certainly do fire up residential proxies, no matter what the site, no matter how useless or duplicate the data which they're trying to get. Not at first - they start by trying to get your content from cheap data center connections - but as soon as some kind of bot-mitigation appears, they move to residential proxies to try and evade that, with a first tier coming from "global south" residential proxies, and then scaling up to (presumably more expensive / less widely available) proxies from the USA (I've seen some from Europe, but very few in relative terms). Each tier also appears to have the option to run JavaScript.
zuzululu 19 hours ago|||
i know a few very large startups that used it to fake their way into an exit

unethical yes but really raises the question as to what we see is real or not

morkalork 19 hours ago|||
Money is real. DAU that don't pay subscriptions, or don't lead to paid conversions on hosted ads, are worthless.
BoorishBears 18 hours ago|||
"Raises the question of what we see is real"

No they really don't, dishonest founders do that.

You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.

I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.

zuzululu 15 hours ago||
ive seen unscrupulous founders fake traction during diligence, which is my day job

but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks

then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection

BoorishBears 15 hours ago||
[flagged]
llbbdd 13 hours ago|||
> forced lowercase

??? shift is an extra key to press

BoorishBears 12 hours ago||
https://www.businessinsider.com/lowercase-typing-altman-dors...

most shibboleths are subtle like that

Chu4eeno 11 hours ago|||
aseigo was the OG, «aseigo: the triumphs and travails of a shift-key-challenged KDE hacker» was his blog byline a couple of decades ago.
llbbdd 12 hours ago|||
I'm...not old enough to read a Business Insider article about being annoyed by lowercase letters.
BoorishBears 2 hours ago||
I think most young people (with friends to text) have already encountered this.
ValentineC 15 hours ago||
From the article:

> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.

I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?

Aachen 6 hours ago||
Opnsense has a traffic capture feature in the interface diagnostics menu, if you want to spot check what servers the devices are currently talking to.

Should be pretty obvious: client devices and internal services will have no traffic >95% of the time, just NTP for timekeeping, DHCP lease renewal, and associated ARP (running total: two dozen packets if you monitor them for a full 24h), then any system updaters (readily identifiable by the initial DNS requests), and finally of course you'll see the traffic of the service that the device hosts, if any, which can be easily dismissed by not looking at incoming connections (scraping uses outgoing connections)

gucci-on-fleek 14 hours ago||
> Tracking which devices still send lots of data when no one else is using the network?

That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).

[0]: https://github.com/jow-/nlbwmon

[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...

[2]: https://docs.mossdef.org/adblock-fast/

arjie 18 hours ago||
What a pity. Mostly I just want personal archives of things so that I can search them much faster than commercial solutions and the like.
klamann 9 hours ago||
I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fair trade-off. Unsurprisingly, I have nothing but negative associations with their mascot.

The FSF has the right idea about all this:

> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.

> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.

https://www.fsf.org/blogs/sysadmin/our-small-team-vs-million...

InsideOutSanta 8 hours ago|
I think all of these mitigations are unfortunate. They hurt one of the things that makes the web cool: it's a stable, stateless, idempotent way to access data.

This makes it a prime target for aggressive scraping by LLM companies, but it also makes it accessible and fast, and a prime target for benign use (like archive.org or "read later" services).

For my own sites, I'll eat the cost of the crawlers (mitigated by making the sites as efficient as possible) and keep them available to everyone.

WarOnPrivacy 16 hours ago||
https://archive.fo/PAcF5
CodesInChaos 6 hours ago||
How much does routing traffic though residential proxies cost?
BLKNSLVR 11 hours ago||
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?

The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").

> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.

Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.

In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:

brdtnet.com

luminatinet.com

bright-sdk.com

luminati.io

As listed in this article, on HN's front page 34 days ago: https://news.ycombinator.com/item?id=48422993 (https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...)

andai 2 hours ago||
Well the argument appears to be, people put them in their apps instead of ads. (Or more likely on top of ads.) The argument is money.

The users presumably don't know about this, or you know, they clicked, "I agree."

Nearly Half of LG Smart TV Apps Contain Residential Proxies

https://news.ycombinator.com/item?id=48635954

andai 2 hours ago|||
Of course the issue with blocking residential IPs is that then they would be prevented from doing normal things on the internet.

At which point, millions of people will be forced to complain to their local representatives and... hey presto? :)

inigyou 6 hours ago||
Why would anyone who doesn't have a use for a residential proxy have an argument for residential proxies?

I use them to scrape closed sites to make the information more open. For example YouTube.

phendrenad2 2 hours ago||
Ever since bots became a problem on the internet 10-20 years ago, it has seemed like the common-sense solution is some kind of micropayment. Pay $0.01 to view the page. When money is on the line, scrapers are likely to be more well-behaved, even if they do pay. The problem is, and has always been, the friction of payment. How do you pay $0.01? The credit card processors will tack on a $6 surcharge. We need a trusted third-party that turn money into "internet article credits" that you can spend in small increments, like a video game. But I suspect that thousands of people have already though of this system, and tried it, but ran into some roadblock. I'm guessing there's some egregious regulation that makes micropayments impossible.
RetroTechie 45 minutes ago|
> I'm guessing there's some egregious regulation that makes micropayments impossible.

More likely there isn't any kind of universal standard that's easy to implement for browser makers, has low overhead, and preserves internet users' anonymity as much as possible.

The currently existing friction of using micropayments is the problem here, I suspect.

andai 18 hours ago||
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.

Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?

Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)

Aachen 5 hours ago||
Blocking in ipv6 works roughly the same way as in ipv4, just that the scale is different. Instead of blocking something like a company's /24 or an ISP's /16 when they don't respond to abuse messages, you block the company's /48 or the ISP's /32. It'll vary per organisation how large a range they got exactly but you can see that in WHOIS. End users are no longer at a /32 (v4) but at /64 (v6), or some prosumers might have a /29 (v4) and /56 (v6). Same concept, just a different prefix length
dylan604 15 hours ago||
> do you want random IoT devices talking to your web server?

Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing

andai 2 hours ago||
Yes that was my point. We should just block all of them.
627467 16 hours ago|
Has no one noticed their miniflux instance failing to fetch feeds because of this?
aendruk 3 hours ago|
So far I’ve only encountered one site that blocked automated access to its web feed. I assume that was just an oversight.
More comments...