Posted by petercooper 15 hours ago
2 access.log [1]
40 botpoop.log [2]
Unfortunately, the best practice is to make css cacheable so you need to keep long histories.
104 access.log
366 botpoop.log
54 access.log
181 botpoop.logAre you speaking from experience, or inferring from articles like this?
I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.
Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.
I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.
I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.
There are lots of solutions to this if you actually need to serve a lot of bytes, but you can get stung even with very simple deployments. Not everyone wants to put Anubis or Cloudflare in front of their site.
The next tier up from free is $25/month or $240 per year.
https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/free/
I made this tool to try and make it easier for beginners to get something up on the public web for free: https://weejur.com
But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.
The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.
I have frequently have had to update a GitHub page, push the change, and then GitHub’s actions puke instead of deploying the change. The workaround is that I have a .txt file with a list of GitHub actions which failed, and when GitHub actions fails, I update that .txt file and push the updated site, which GitHub actions will hopefully successfully deploy.
GitHub pages are OK for pages which aren’t updated very frequently, but they are not OK for pages which update frequently.
If you can't, well then... We're stuck.
Because you have no CDN, all the bots are coming directly to you.
Because that's no CDN or distributed WAF, they're hammering your box.
Because your IP is directly exposed, you're also showing up on Shodan.
Your server may have enough CPU and RAM to handle the load, but does your monthly bandwidth allocation with your provider?
Many are finding previous bandwidth limits (starting around 10TB/month) are no longer enough.
My single static webpage with no updates in 3 years is doing that, which is (one of the reasons) how I end up where that site (and many others in business and personally) is.
You're chasing a dream for a world that doesn't exist anymore.
“For a low $/GB, we’ll give you everything from this site and 1000 others as (structured data/a database)!”
(yes there are lots of good counter arguments to this, but before you reply think ahead a couple extra steps)
This is a small booking app without any useful information at all, it surprises me that the AI boots have no discernment about what the are scraping, just wasting their own and other peoples resources. And their own reputation! You would thing they could spare a few tokens on a classifier model to do a quick evaluation of their scraping efforts, but apparently they do not.
Anyway, I have done my best to block these UAs and so far it seems to have improved the situation.
https://developers.cloudflare.com/bots/additional-configurat...
my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.