Posted by damir 10/23/2024
Ask HN: Website with 6^16 subpages and 80k+ daily bots
It's little over a year old and now every day there's 60k-100k bots visiting and crawling the shit out of two-trillion-something sub pages...
I am out of ideas what to do with this site. I mean, it's probably one of the largest websites on the Internet, if counted by sub-pages...
What cool experiment/idea/stuff should I do/try with this website?
I'm sure AI could be (ab)used somehow here... :)
Alternatively, sell text space to advertisers as LLM SEO
Not sure it helps in SEO though...
How important is having the hex color in the URL? How about using URL params, or doing the conversion in JavaScript UI on a single page, i.e. not putting the color in the URL? Despite all the fun devious suggestions for fortifying your website, not having colors in the URL would completely solve the problem and be way easier.
You could try serving back html with no links (as in no a-href), and render links in js or some other clever way that works in browsers/for humans.
You won’t get rid of all bots, but it should significantly reduce useless traffic.
Alternative just make a static page that renders the content in js instead of php and put it on github pages or any other free server.