Top
Best
New

Posted by Bender 3 hours ago

How to Block Some of the Bots(nochan.net)
53 points | 38 comments
CqtGLRGcukpy 1 hour ago|
If you are unable to read this, there is an archived copy at https://archive.ph/d3236
ACCount37 4 minutes ago||
I couldn't. Amusing that archive.ph's own crawler, evidently, went through just fine.
Bender 1 minute ago||
I disabled blocking for a few hours then crawled several of my pages with their site.
Barbing 52 minutes ago|||
Thanks. Regretful implementation*, being a normie on iOS Safari:

https://i.ibb.co/vCDH79d0/IMG-0303.png

…and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.)

*edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)

Bender 47 minutes ago||
That makes sense. I block most data-centers and that would include much of Apple's data-centers. What I am doing is of course unorthodox for a blog. This is more of a test site to show what could be done. People can pick and choose which features they like for their particular use cases.

I could some day split it out into a blog and a demo site.

hdjrudni 1 hour ago||
I wonder why you'd be unable to read it? :-)
cwillu 44 minutes ago||
Because of shitty over-zealous blocking, probably.
userbinator 15 minutes ago||
You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.

Block on traffic volume and request frequency if that's causing a problem.

(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)

Bender 6 minutes ago|
I can accept that. I've sat with RMS a few times. He's an interesting and very smart person. There are things he and I agree and disagree on. I'm sure I would never hear the end of it on this one. I'm glad he put the cancer into remission and will have many many more years.
fxtentacle 2 hours ago||
I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
Bender 2 hours ago|
When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.
boznz 21 minutes ago||
I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.
binaryturtle 2 hours ago||
410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks!

Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)

gruez 2 hours ago||
I didn't even make it that far, I got PR_END_OF_FILE_ERROR which indicates it couldn't even get past the tls handshake.
ErroneousBosh 2 hours ago||
I mean that just might be the dying gasp of a puddle of scorched fibreglass and boiling copper where the server used to be.
Bender 2 hours ago||
Mostly harmless(c)

    load average: 0.00, 0.01, 0.00
Bender 2 hours ago||
Real browsers send it. [1] Some of the reader apps do not. Most bots aside from those utilizing Chrome Headless do not.

People can see a few headers here [2]

[1] - https://caniuse.com/?search=sec-fetch-mode

[2] - https://nochan.net/.env

juleiie 2 minutes ago||
The guy is certainly kind of visibly ravaged by the more shady denizens of the global internet.

But there are some fun things to read there in any case.

genodethrowaway 58 minutes ago||
and all valid traffic too, judging by these HN comments (and my own attempts to connect).
Bender 55 minutes ago|
The comments can be misleading. About 2400 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
fatty_patty89 45 minutes ago||
what's up with those response headers? "adult" ...

"ai" ...

response length 68

Bender 43 minutes ago|
AI will not ingest or operating on anything with obscene or foul terms or at least that is what I have been led to believe.
fatty_patty89 41 minutes ago||
weird misdirection excuse

edit:

>"I can not see votes or karma and do not require social validation. "

>proceeds to downvote

classic

Bender 35 minutes ago||
I did not and could not downvote you as you replied to me. Only others can. I have not downvoted any comments on this thread. I appreciate all feedback including yours.
RobotToaster 1 hour ago||
Blocks firefox's built in VPN.
Bender 1 hour ago|
That's likely from me blocking data-centers as they probably do not have residential exit nodes. There are a couple nftable rules that could block most VPN's as well but they also block some cellular networks so I left that out.
FabCH 55 minutes ago|
The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive.

The added explanations by the author in this comment thread are hilarious. You sir are a good writer.

Bender 16 minutes ago|
Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.
More comments...