Top
Best
New

Posted by damir 10/23/2024

Ask HN: Website with 6^16 subpages and 80k+ daily bots

Last year, just for fun, I created a single index.php website calculating HEX colors to RGB. It takes 3 and 6 digit notation (ie. #c00 and #cc0000) and converts it to RGB value. No database, just single .php file, converting values on the fly.

It's little over a year old and now every day there's 60k-100k bots visiting and crawling the shit out of two-trillion-something sub pages...

I am out of ideas what to do with this site. I mean, it's probably one of the largest websites on the Internet, if counted by sub-pages...

What cool experiment/idea/stuff should I do/try with this website?

I'm sure AI could be (ab)used somehow here... :)

287 points | 201 commentspage 3
ericyd 10/24/2024|
For the purpose of this post, are we considering a "subpage" to be any route which can generate a unique dynamic response? It doesn't fit my idea of a subpage so wanted to clarify.
simne 10/24/2024||
As addition to already mentioned robots.txt and ideas of penalties for bad bots (I especially like idea of poisoning LLMs).

I would be add some fun with colors - modulate them. Not much, I think would be enough to change color temperature to warm or cold, but same color.

Content of modulation could be some sort of fun pictures, may be videos for most active bots.

So if bot put converted colors to one place (convert image), would seen ghosts.

Could add some Easter eggs for hackers - also possible conversion channel.

ipaddr 10/24/2024||
Return a 402 status code and tell users where they can pay you.
danybittel 10/23/2024||
https://www.youtube.com/watch?v=fwJHNw9jU_U
stuaxo 10/24/2024||
If you want to mess with bots there is all sorts of throttling you can try / keeping sockets open for a long time but slowly.

If you want to expand further, maybe include pages to represent colours using other colour systems.

berlinbrowndev 10/24/2024||
Isn't this typical of any site. I didnt know it was 80k a day, seems like a waste of bandwidth.

is it russian bots? You basically created a honeypot, you out to analyze it.

Yea, AI analyze the data.

I created a blog, and no bots visit my site. Hehe

dian2023 10/24/2024||
What's the total traffic to the website? Do the pages rank well on google or is it just crawled and no real users?
nitwit005 10/24/2024||
You're already contributing to the world by making dumb bots more expensive to run.
is_true 10/23/2024||
You could try to generate random names and facts for colors. Only readable by the bots.
OutOfHere 10/24/2024|
Have a captcha. Problem solved.

I highly advise not sending any harmful response back to any client.

More comments...