LLM are the best way to read the web. right to the point, no ADS

This is like technological cannibalism.

I stopped posting to my website. Why should it be so much work to stop this theft?

Would it be helpful to have geofencing and regulation?

I'm working on a web app right now, with the intention of it going to be 100% paywalled. It's 95% complete, but the remaining 5% is just implementing the paywall. In the meantime, the app is live and operational with a fully functional signup. I am deleting about 100 new bot signups per day right now, it is crazy out there.

I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.

> My visitor stats got so polluted I couldn't trust my own numbers. This was the one that hurt. I'm trying to run a business here. I want to know what real people read on my site so I know what to build next. I couldn't see them through the bots.

Are bots the solution to nosy websites that want to know things about their visitors?

I'm surprised that the solution wasn't to poison the data so they will train on junk data

The annoying part is a large percentage of misbehaving bots (not obeying robots.txt for example) are via end user proxies across the world. However most of these aren't doing full-browser loop, so if you are behind cloudflare, you can do non-interactive challenge and that can help quite a bit.

Original HN title: "99% of My Website Traffic Is Bots"

Not clear how the author arrived at the precise 99% figure; perhaps "99%" is a figure of speech

"And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers."

"I'm trying to run a business here."

What's the business

(where "business" is defined as "buying and selling")

From https://patronview.com/robots.txt

   # As a condition of accessing this website, you agree to abide by the following
   # content signals:
   
   # (a)  If a Content-Signal = yes, you may collect content for the corresponding
   #      use.
   # (b)  If a Content-Signal = no, you may not collect content for the
   #      corresponding use.
   # (c)  If the website operator does not include a Content-Signal for a
   #      corresponding use, the website operator neither grants nor restricts
   #      permission via Content-Signal with respect to the corresponding use.
   
   # The content signals and their meanings are:
   
   # search:   building a search index and providing search results (e.g., returning
   #           hyperlinks and short excerpts from your website's contents). Search does not
   #           include providing AI-generated search summaries.
   # ai-input: inputting content into one or more AI models (e.g., retrieval
   #           augmented generation, grounding, or other real-time taking of content for
   #           generative AI search answers).
   # ai-train: training or fine-tuning AI models.
   # use:      how AI systems may consume the content (immediate, reference, or full).
   
   # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
   # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
   # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
   
   # BEGIN Cloudflare Managed content
   
   User-agent: *
   Content-Signal: search=yes,ai-train=no,use=reference
   Allow: /

Perhaps this could be construed as a license, e.g., permitting or prohibiting certain uses of the "content"

If, for example, the website operator had enforceable intellectual property rights in the "content", such as copyrights, then perhaps the operator could restrict access to the "content" under the threat of litigation to enforce those rights

Basic questions

1. Is the "content" protected by intellectual property rights, e.g., copyrights

2. Does the website operator have intellectual property rights in the "content", e.g., copyrights

3. Does the website operator have agreements with the rights holders, e.g., granting the operator authorization to restrict access to the "content"

I don't look at the logs of my personal site very often as it is a static site, but just went to check, and yeah, it's almost all ai crawlers. Note sure what is going on here, but I hope this isn't really anthropic:

  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /credentials.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.yml HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /key.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /config/.env HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /service_account.json HTTP/2.0" 404 366 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /serviceAccountKey.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-adminsdk.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /Dockerfile HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.github/.env HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.docker/config.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.npmrc HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.boto HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.s3cfg HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.svn/entries HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.htpasswd HTTP/2.0" 404 346 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /terraform.tfstate HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /docker-compose.yaml HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.vscode/launch.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_rsa HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ed25519 HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ecdsa HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/authorized_keys HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/known_hosts HTTP/2.0" 404 343 "-" "anthropic-ai"