All of these solutions seem expensive, if you're paying for outbound bandwidth.

I've thought about tying a hidden link, excluded in robots.txt, to fail2ban. Seems quick and easy with no side-effects, but I've ever actually gotten around to it.

I had to follow a link to see an example:

"A glass is not impossible to make the file and so deepen the original cut. Now heat a small spot on the glass, and a candle flame to a clear singing note.

— context_length = 2. The source material is a book on glassblowing."

I think random text can be detected and filtered. We need probably pre-generated bad information to make utility of crawling one's site truly negative.

On my site, I serve them a subset of Emergent Misalignment dataset, randomly perturbed by substituting some words with synonyms.

It should make the LLMs trained on it behave like dicks according to this research https://www.emergent-misalignment.com/

Really cool. Reminds me of farmers of some third world countries. Completely ignored by government, exploited by commission brokers, farmers now use all sorts of tricks, including coloring and faking their farm produce, without regard for health hazards to consumers. The city dwellers who thought they have gamed the system through high education, jobs and slick-talk, have to consume whatever is served to them by the desperate farmers.

Hope you don't mind if I point out a couple of small bugs in babble.c:

1. When read_word() reads the last word in a string, at line 146 it will read past the end (and into uninitialised memory, or the leftovers of previous longer strings), because you have already added 1 to len on line 140 to skip past the character that delimited the word. Undefined behaviour.

2. grow_chain() doesn't assign to (*chain)->capacity, so it winds up calling realloc() every time, unnecessarily. This probably isn't a big deal, because probably realloc() allocates in larger chunks and takes a fast no-op path when it determines it doesn't need to reallocate and copy.

3. Not a bug, but your index precomputation on lines 184-200 could be much more efficient. Currently it takes O(n^2 * MAX_LEAF) time, but it could be improved to linear time if you (a) did most of this computation once in the original Python extractor and (b) stored things better. Specifically, you could store and work with just the numeric indices, "translating" them to strings only at the last possible moment, before writing the word out. Translating index i to word i can be done very efficiently with 2 data structures:

    char word_data[MAX_WORDS * MAX_WORD_LEN];
    unsigned start_pos[MAX_WORDS + 1];

(Of course you could dynamically allocate them instead -- the static sizes just give the flavour.)

word_data stores all words concatenated together without delimiters; start_pos stores offsets into this buffer. To extract word i to dest:

    memcpy(dest, word_data + start_pos[i], start_pos[i + 1] - start_pos[i]);

You can store the variable-length list of possible next words for each word in a similar way, with a large buffer of integers and an array of offsets into it:

    unsigned next_words[MAX_WORDS * MAX_LEAF];     // Each element is a word index
    unsigned next_words_start_pos[MAX_WORDS + 1];  // Each element is an offset into next_words

Now the indices of all words that could follow word i are enumerated by:

    for (j = next_words_start_pos[i]; j < next_words_start_pos[i + 1]; ++j) {
        // Do something with next_words[j]
    }

(Note that you don't actually store the "current word" in this data structure at all -- it's the index i into next_words_start_pos, which you already know!)

Does this really work though? I know nothing about the inner workings of LLMs, but don't you want to break their word associations? Rather than generating "garbage" text based on which words tend to occur together and LLMs generating text based on which words it has seen together, don't you want to give them text that relates unrelated words?

Why not show them ads? Endless ads, with AI content in between them?

Maybe a dumb question but what exactly is wrong with banning the IPs? Even if the bots get more IPs over time, surely storing a list of bans is cheaper than serving content? Is the worry that the bots will eventually cycle through so many IP ranges that you end up blocking legit users?

A thing you'll have to watch for is these agents actually being a user's browser, just the browser provider is using them as a proxy.

Otherwise, there are residential IP proxy services that cost around $1/GB which is cheap, but why pay when you can get the user to agree to be a proxy.

If the margin of error is small enough in detecting automated requests, may as well serve up some crypto mining code for the AI bots to work through but again, it could easily be an (unsuspecting) user.

I haven't looked into it much, it'd be interesting to know whether some of the AI requests are using mobile agents (and show genuine mobile fingerprints)

I suppose once you've lured them into reading a couple garbage pages you've successfully identified them as bots. You could then serve them garbage pages even for real urls as well just in case they ever got smart enough to try and back out of endless garbage. You could probably do a bunch of things that would only affect them specifically to increase their costs.