I recently got a surge from Singapore and heard a lot more peeps in my circle of friends saying the same thing...

I think it’s wild how AI and data bots are putting certain business models under pressure. We’ve already seen the same thing happen with Tailwind.

I visited someone's blog and didn't have to solve a cloudflare challenge! That's amazing work

I wonder what the web would be like if we priced bandwidth at the requester point and as you go.

Cloudflare's "Verify you are human" captcha is the new cookie banner.

at this point the web is mostly machines politely asking other machines for permission to read each other's content, and humans are the ones triggering the captchas.

An interesting article, and an interesting problem to have!

The problem, to recap the title, is that "99% of My Website Traffic Is Bots".

How to fight all of those bots, all of those web-scrapers...

If the sole underlying issue is bandwidth for one's website (someone doesn't have enough of it and/or they pay too much for it), then one possible solution to this might be for a single tech company to create an online cache of web pages, accessible to all bots as an alternate route to fetch the desired web pages.

If a tech company (or consortium thereof) decided to do this, they could certainly charge money to each AI / bot utilizing the service...

Which means that it might be an investable idea as a for-profit service...

Now, if the underlying issue is the privacy of individual web pages (or relative privacy, as the case may be!) from the public internet, then perhaps the solution is to simply put those behind user logins and/or a gauntlet of tests (which could also be tied to login!) designed to exclude AI/bot web traffic from human web traffic.

Consider what some BBS'es and the online services of yesteryear used to do (Compuserve, The Source, Prodigy, QuantumLink, AOL, etc.)... basically they'd get a user's mailing address, and then physically snail mail them their login and password to their physical home address...

Oh sure, user registration wasn't as fast back then as it is today... it might take several days to receive your username and password via postal mail -- but as a system operator you were 99.999999999% guaranteed that when you saw such a login on your BBS, that it was an actual live human user...

There may be a market for a service like that, too...

That is, validate that actual users are actual users, and give them some kind of credential that can be checked by an actual web site... find a way to do this at scale...

Anyway, just thinking aloud... a very interesting article, and a very interesting problem to have!

Sigh .. same here. I don't write blog posts often (enough) but the ones I do write are from personal experiences and I take a lot of care with them. I look at my logs snd see bots everywhere, but now I just let them get on with it. AI scrapers are different though; they get to read my content which, just for them contains a smsttering of finest digital toxin. A pox on your datasets!

Did he really say that the VPS would be in trouble if it received 5 requests per second of mostly static pages? Something that currently costs $90 per month to serve through CloudFlare?