Posted by raybb 3 days ago
The module doing proof of work might be a part of your frontend that is static and on CDN.
If you want agents still have access to your data, just not through scraping, provide alternative format for them. Like periodic sqlite dump.
I made a prototype of such PoW two years ago. A challenge is a random number. Checking the client's solution is calculating one hash of the challenge glued to client's solution and checking if the resulting hash is less than a specific value. The lower the value, the harder the task for the client to do. Client needs to calculate this hash multiple times to find a response that will result (when glued with challenge) with low enough hash.
The server needs to keep short lived cache with challenges so the client can't just make up one or reuse another solution and provide two responses for every challenge, first, very small and fast with a challenge, and the second one with actual content (if response was correct). Nothing much worse than CORS preflight.
Most web proof-of-work stuff has worked because it breaks the normal flow and the typical scraper is supremely dumb, not because of the proof of work, which has largely been a total red herring.
The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.
Like there's that anime girl who harasses you when you visit so many open source web sites that just continues the pattern of being harassed by cookie dialogs and consent dialogs and pop-up ads and being asked for your email address and of course there is Wikipedia which is doing more and more of the same.
Another web is possible but web populists don't really believe it.
Just to get started, bots have been better at filling out CAPCHTAs than humans for a long time. I knew someone who got his site hacked by Russians circa 2008 because he published a neural network in freakin PHP that could read them. Anubis causes humans stress, slows down task completion, probably raises peoples blood pressure, and cumulatively causes a great deal of harm to users the same way Cloudflare's system does. Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
CAPTCHAs, cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits with triple damages.
Hence why my solution is, ironically, micropayments. That would be so cheap that any human could even access thousands of webpages per day for pennies (or however expensive sending a signal across wires is…it can’t be much), including shared dumps of scraped data from billions of pages.
And Anubis: it is uncomfortably close to erotica, in a way that makes me panic when it pops up while I'm at the office. I wouldn't care if it was a picture of a puppy or something, but not an anime catgirl.
Equating bot prevention measures with “please give us permission for 1,000s of companies to stalk you around the Internet, no? Here are some dark patterns to try make you accidentally agree” is rather unfair IMO. They are an inconvenience but for the protection of the sites rather than for the betterment-of-adtech-via-conning-you-the-viewer-with-malicious-compliance.
> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit
They could, but given these methods seem to work currently most (almost all?) are not. They will start doing so if the amount of resources protected in this way increases to the point where it is worth it, or course, that is why memory-hard processes are being worked on which will make that route less practical.
> cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits
I can't disagree there, though that would be quite an inefficient process and requires a bunch of money up front from who-ever champions the legal challenge, so I doubt it would actually be at all practical.
Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
> Another web is possible but web populists don't really believe it.
How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?
The implication is that while they are waiting for the async loop to complete for one site, the bot can be scraping others that have already had the challenge completed (usually the one challenge stays in force for some time afterwards) or that have no such challenge.
Memory hard challenges will reduce the effectiveness of this, but not remove it completely.
Though that these methods are working at all implies that most (almost all?) bots are not bothering to use such async methods as there are sufficient resources not protected by them that this complication is not yet worth implementing.
Sharing cookies along a strata of inter-team bots is one trick.
_
Web scraping is (among the other pieces) also for the more technically inclined among the "little people" to have computers do things that people they don't have the money to hire might do. Making web scraping more difficult, which won't stop the big players who will work around the hurdles one way or another, is "pulling up the ladder".
_
> What alternative web could change this state of things?
A different architecture for trafficking web contents, one that doesn't need the "shield" of something like Cloudflare or "tar pit" of something like Anubis in the first place.
IPFS is or is something like one piece of that picture but there are other essential pieces to complete that picture.
I know I'm incensed at all of this because I feel like I'm the one that endlessly waits in front of the Anubis challenge for 5-10 seconds, or keeps getting extremely difficult or brain-draining hCaptchas thrown at me with no real solutions.
And I'm stuck with low-end compute for probably the rest of time at this point - not a single machine in the house has more than 8GB of RAM. So I'm just doomed to wait and burn my time and my sanity some more.
Ya know, on the one hand, I wonder how many federal officials own nVidia stock and stock of other AI companies...
But then again, on the other, I wonder how many nVidia employees (and those of other AI companies!) own government T-bills (and other government backed securities!) in their 401K or other investment accounts!
You know, everybody in large corporations and governments owning shares in other large corporations and governments -- adds a whole new level of meaning to "stakeholder capitalism"... :-)
(In the far future, there'll probably be small foreign governments which become their own ticker symbol -- and get publicly traded on various stock exchanges! :-) You know, like: "Would you like to buy a share in Denmark (DNMK) or perhaps Norway (NRWY)? I hear they're very competitive with Apple (AAPL) right now! Great price to earnings (PE) ratio!" :-) )
I am simplifying terribly but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
An interesting point! Yes, that is definitely one way that currency could be looked at...
>The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
Agreed -- but if we were really to analyze things, scrutinize the way things really work at the highest degree of resolution, then we would have to observe that:
a) There are multiple countries/currencies in the world.
b) That each country's/currency's inflation rate is different than all of the other countries/currencies.
c) If each country's/currency's inflation rate is different, this means that each countries's issuance of new shares (and control of interest rates) -- their financial policies, are, by comparison, either more effective, on par, or less effective than those of other countries used for comparison...
Phrased another way:
All countries do not control the rate of inflation of their currencies equally.
Phrased another way:
Some nations are better at controlling their currency's rate of inflation than other nations.
True or not true?
>I am simplifying terribly
As am I! :-)
>but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
Again, an interesting point!
And yes, I could agree with that!
Manifestly true, for lots of reasons.
The analogy only goes so far because currency is traded for goods, services, and debt all the time whereas shares are usually just traded for currency.