Posted by damir 4 days ago
Ask HN: Website with 6^16 subpages and 80k+ daily bots
It's little over a year old and now every day there's 60k-100k bots visiting and crawling the shit out of two-trillion-something sub pages...
I am out of ideas what to do with this site. I mean, it's probably one of the largest websites on the Internet, if counted by sub-pages...
What cool experiment/idea/stuff should I do/try with this website?
I'm sure AI could be (ab)used somehow here... :)
Alternatively, sell text space to advertisers as LLM SEO
Not sure it helps in SEO though...
How important is having the hex color in the URL? How about using URL params, or doing the conversion in JavaScript UI on a single page, i.e. not putting the color in the URL? Despite all the fun devious suggestions for fortifying your website, not having colors in the URL would completely solve the problem and be way easier.
You could try serving back html with no links (as in no a-href), and render links in js or some other clever way that works in browsers/for humans.
You won’t get rid of all bots, but it should significantly reduce useless traffic.
Alternative just make a static page that renders the content in js instead of php and put it on github pages or any other free server.
I would be add some fun with colors - modulate them. Not much, I think would be enough to change color temperature to warm or cold, but same color.
Content of modulation could be some sort of fun pictures, may be videos for most active bots.
So if bot put converted colors to one place (convert image), would seen ghosts.
Could add some Easter eggs for hackers - also possible conversion channel.