Top
Best
New

Posted by surprisetalk 4 hours ago

The 100k Whys of AI(lcamtuf.substack.com)
107 points | 56 comments
TrackerFF 1 minute ago|
What is worse, IMO, is that these GenAI books have found their way into physical stores. You know, the few that are still left.

I've found AI slop at many big box stores (think Walmart, Target, etc. and all their equivalents around the world) - which I suspect are "industry plants", meaning that the publishing house will have someone internally generate books like these, and sell them as physical copies around the thousands of stores I mentioned.

It is the equivalent of record labels pushing their own in-house GenAI artists.

dlenski 3 hours ago||
A nice illustration of the homogeneity of LLM responses. Another way to describe this effect would be…

If you ask humans to write 1,000 books, you're asking 1,000 different humans with different experiences and different skills and different moods (etc.) to write those books.

But if you ask LLMs to write 1,000 books, you're probably only talking to 3 or 5 different models, tops. And they've all trained on the same or similar data, and are trained to respond in very similar ways.

The LLMs don't differ much in anything like "life experience" or "skills", and they don't really have anything like a "mood" independent of the prompts you've given them.

amelius 26 minutes ago||
I don't think the comparison to humans works. It is as if you expect that we can easily train many different LLMs to solve the similarity problem, but that is far from guaranteed.
NitpickLawyer 1 hour ago|||
> A nice illustration of the homogeneity of LLM responses. [...] And they've all trained on the same or similar data, and are trained to respond in very similar ways.

I mostly agree, but this is a very simplified explanation. The models are indeed trained to respond in similar ways, for "basic" prompts. And that's as much a feature as it is a bug. In other words, the bug becomes apparent only if you give 100+ basic prompts. But giving it 100+ basic prompts and expecting originality is a silly endeavour. That's not how you get originality.

The way I'd go about to generate 1000 books, while expecting different outcomes is something along these lines (and nowadays you can ask your favorite LLM to wire up this workflow for you, with decent outcomes):

1. Ask for a list of 20 features that define a book (genre, style, number of characters, tropes, plot, continuity, relationships, etc.)

2. For each feature, ask for a list of 50 examples, ordered from most common to the most unique.

3. Randomly pick 10 features, and for each pick one of the 50 generated items. Ask for the rest of the features to match the theme.

4. Ask for 10 possible book outlines that match the chosen features, randomly pick between 2-8.

5. Create a detailed prompt that includes all the above features, and ask for a synopsis for each chapter, given the above outline chosen.

6. Given {features} and {outline} and {synopsis} write chapter 1.

7. for each chapter in list, given {...} and (optional) previous matching chapter(s), write chapter n+1

(optional 8.) given {...} and 2-3 consecutive chapters, align the ending / beginning of a new chapter for style / features / continuity, etc.

(optional 9.) given {...} and the whole book, list chapters / paragraphs that don't match the given {...} and provide a list of 5 improvements. (randomly choose 1 and ask for an edit).

----

Now, this probably won't give you something like cloud atlas, but they'll at least be different books. That's how I'd do it if I wanted to see how different they can write. Not 1000 "basic" prompts and expecting originality.

noduerme 1 hour ago||
That whole thing would get you 1000 variants of existing art. But if you asked a thousand different designers to do a cover for the same book...
NitpickLawyer 57 minutes ago||
> 1000 variants of existing art.

This is very naive. I can almost guarantee that some combinations of 20 * 50 features will hit on something that has never been written before in that specific combination. And if that's still not enough, increase the number of features. Add more randomness, add more steering, add random steering in random chapters, change it up, and so on.

noduerme 19 minutes ago|||
I'm an art director. Finding a sequence that hasn't been hit in that specific combination is not sufficient to justify paying someone $150 an hour to go be creative.
spwa4 14 minutes ago|||
> Add more randomness, add more steering, add random steering in random chapters, change it up, and so on.

That doesn't work for AI models. The whole training process depends on the basic principle that if you take the average of 100, in this case book cover designs, that the average is less like randomness than any individual cover you've used to make your average.

So the output will, by necessity, be closer to the average.

The human learning algorithm is much, much more data efficient than models. A absolute top human expert will have read/seen/heard/talked/... about 160 million "tokens" (that's about 2000 books). Frankly, the nerve inputs of all experiences of an entire human life, from baby to rewriting relativity theory, are only a couple dozen gigabytes.

Qwen 3.6 27B has been trained (as in seen ~10 to ~50 times) 8 trillion tokens, or to put it another way: for every second you will have spent "gathering life experiences" (ie. your whole life) on your deathbed Qwen 3.6 27B has spend about 50.000 seconds learning. And really that figure should be multiplied by the 10 or 50 training iterations.

Add another 3 or so orders of magnitude and you've got ChatGPT. By this measure, the human brains outperforms ridiculously overspecced ML models (because that's what ChatGPT and the like are) in efficiency a factor of by 5 million or more. This is the reason humans are still faster than ML models.

As for human training iterations: we can be simple: it's 1. In fact, it's impossible to make it even 2. Of course, when it comes to human performance: we are a better but not fundamentally different version of genetic algorithms. Do most humans perform? The honest answer is no. 1 in 1000, and that's very generous, improves SOTA. You absolutely need the 1000 failures though, as anyone whose tried a PhD (or even just design a large program) knows.

So we are very far away from allowing AI models to do what humans can do: take one example and produce, from one example, a better output. And there will always be much more variation in that approach. But ... most human attempts to do something are total crap. Most AI attempts to do something will succeed, but they'll be comparatively be bland, tasteless, "without soul", ...

And this is ignoring the problem that AI also has a massive limitation (that can't be solved, no matter how many nvidia cards you have) in that it trains against historical data. And counterfactuals don't work. What would have happened had Shakespeare decided Macbeth's wife was a force for good? Would the king still get murdered? Would it still be a great story? You can't work with counterfactuals.

smusamashah 2 hours ago|||
Reminds of Pluribus.
bigbangcmbr 1 hour ago||
Pluribus is kinda different. An LLM cannot wander too far from the average. Even if it wanted too. In pluribus, the 'others' work toward a common goal, each utilizing their own expertise, knowledge and experiences in a shared way to achieve a common goal. Each is unique. They can, if they want, perform as the host's individual before the the joining. To put it other way, the other in pluribus are convergent by choice, llms are convergent by design.
throw310822 2 hours ago|||
> you're asking 1,000 different humans with different experiences and different skills and different moods

Simply, if you ask an LLM, you're asking always to the same mind, and always for the first time.

scotty79 2 hours ago||
Also since those are lazy, you are also asking always in the same manner. How homogeneous were the prompts that generated those covers?

People are making cookies with cookie cutter number 5 and other people wonder how come they are all the same.

gmerc 2 hours ago||
Classic self selection effect though - if you’re resorting to LLM writing you’re almost certainly skewing lazy enough to not even bother trying to add perturbations strong enough to make the response deviate from the uniformity of the slop.
ekianjo 1 hour ago|||
prompts will give very different results. this is where you do the work.
cryo32 1 hour ago|||
I disagree. The LLM outputs really do lack anything original or interesting. They just produce banal copy whatever you ask them.

A good editor could probably reduce all LLM outputs on a subject down to the same point.

roncesvalles 1 hour ago||||
Yes but not very different results (unless you're adding new information to your prompt or reducing some ambiguity). Prompt engineering is mostly pseudoscience.
zarzavat 1 hour ago||
What we need is steering so that we can have models with different personalities, not just different prompts (because context is subject to forgetting), but this will never happen with closed-weight models, I'm not sure if it's even feasible at scale.

Yet another reason why the future is open weight.

hansmayer 57 minutes ago|||
[dead]
fragmede 2 hours ago||
that discounts, how much the other context, ie, the system, prompt, and any sort of other context submitted to the model that can affect the output. If you ask a model as a patient for medical advice versus as a doctor, you will get different output from the same model.
firefoxd 3 hours ago||
When you generate one or two blog posts with LLM they look pretty good. And you will be impressed with that one clever bit it adds that you didn't even ask for. But then you generate 50 of them and they all converge into the same pattern. It's hard to prove that an article is AI generated but they are instantly recognizable.

An aside, I usually take my written blog posts through a pass on Notebooklm to generate a podcast like discussion about it. It used to be a good way to extract some insights I haven't thought of. But after 50 of them, I can predict what the host will "pushback" on and exactly when. Then they magically resolve their differences and agree with whatever the idea was. It's truly impressive when you just consume sporadically. But listen frequently and they converge into one blob.

qsera 2 hours ago||
> It's truly impressive when you just consume sporadically. But listen frequently and they converge into one blob.

And something that shows that behavior is a scammers wet dream!

rusk 2 hours ago|||
I suspect there are new invariants emerging. We don’t know what they are and we will probably have to reach into the liberal arts to describe them but to me what you’re seeing is akin to the subatomic world exposing itself through diffraction patterns.
tantivy 2 hours ago||
You're just looking for the study of rhetoric. LLMs have clustered on certain rhetorical patterns/gestures, probably because of a combination of frequency in input and bias in training. But rhetoric also concerns the logical structures that underpin communicative techniques, and it's this logical infrastructure that's shaky or bizarre in LLM content (like the GP noticing how "pushback" always resolves without further examination).
rusk 1 hour ago||
> You're just looking for the study of rhetoric.

I presume you mean, that what I and others is observing is patterns in mere rhetoric. That this is just unimportant window dressing around the actual problem solving.

Yet, generation of rhetoric seems to be one of the key usecases, and one of the key features that makes this technology seem “intelligent”.

smitty1e 1 hour ago||
> they all converge

AI is regression to the mean.

Much like Socialism.

Om an acute basis, AI can be just as helpful as that safety net.

As a chronic matter, "it's not excellence--it's mediocrity".

Foobar8568 37 minutes ago||
And capitalism as seen in the USA is regression to the bottom of the cesspool?
vintermann 3 hours ago||
There used to be a word for this in generative AI: mode collapse. It's not that the model doesn't generate human-like responses, it's that it generates the same 0.0001% of possible human like responses every time. It's almost certainly the instruction tuning which is responsible, maybe some small part of blame could go to the rollout policy (I have no idea how rollout policy works these days).
licnep 2 hours ago||
It's even more worrying when you look at the contents of these "books", they are riddled with erros:

https://infosec.exchange/@lcamtuf/116785283147249092

jdthedisciple 44 minutes ago||
Truly sad state of affairs
JSR_FDED 1 hour ago||
This is atrocious
thinkingemote 2 hours ago||
On HN many comments under many threads are about whether the submission was written by AI. You could say I have noticed a pattern in Hacker News comments!

In these comments there's a common pattern where some users argue that they do not agree that the submission was LLM written and they often focus on specific details to refute it (e.g em-dashes) and some users see the overall pattern clearly that it's totally obvious. For me it's a kind of smell, it's off putting and it's obvious. The article says to "trust your gut". But it's also something that comes with practice and time, it's not some innate thing. People may have better things to do than expend mental energy noticing patterns in a bunch of social media posts. The more I see it, the more I see it.

The take away I get is that it's okay to notice patterns and it's okay to not notice patterns. Remember that other people may be noticing patterns and associations in things that you might miss. Be charitable.

Far more interesting questions are:

1) If you cant see the patterns of LLM writing, does the idea that the thing you liked was written by LLM worry you?

2) If you can see the patterns clearly is the fact that it's LLM written worry you?

Because in our comments there's many who do not care that LLM's are writing content and theres many who do care. But are these correlated with those who can see the LLMs or who are blind to them?

SXX 1 hour ago|
I'm not worried abou LLM written content, my problem is not word prediction. My problem with it pretty much like with mass produced self help books decade ago.

Good human writing especially on highly technical topics its usually compression of information.

Like you have some experience you want to share with others and you work your brains try to put it into concise story.|

Problem us: AI generated texts are opposite 99% of the time: author usually have bullet point list to feed into machine to add hallucinated word predicted story on top of it.

So signal to noise ratio is much worse.

So reading AI texts is pretty much like listening for stories from humans with mental problems - no one really wants to listen to hallutinations even if somewhere inside there is some useful information.

exitb 2 hours ago||
Notably, in programming this is actually a desirable feature for most problems. Even human programmers are taught to produce predictable and obvious code whenever possible. I wonder is ultimately this is an artifact of optimizing the models for code, that they become less creative.
_3u10 2 hours ago||
I’ve rarely experienced this. Typically what is requested is code that has unpredictable pauses, takes unbounded time, has two kinds of null, etc.
rusk 2 hours ago||
Determinism is a desirable property for software, yes, and its lack thereof from LLM’s is a common complaint, but often a feature depending on who you ask. There is an element of randomness, “hotness” that is central to who LLM work but the pattern we see here manifesting reveals the deterministic processes below, but I don’t think you could rely on this technology to be deterministic, if that’s what you wanted.
xandrius 52 minutes ago||
The whole point of the thesis is that because the cover image are very similar, therefore LLMs are bad at writing text?

I think it's that today's LLMs have access to poor/generic image generation models and people find it easier to ask ChatGPT or NanoBanana to make a cover instead of fine tuning a small SD model for the purpose.

Lerc 1 hour ago||
I think for that instance to carry weight you would have to provide evidence that the mosaic of books were the product of different people using AI. If it is just one person doing variations on the same thing then it wouldn't mean very much.
noduerme 1 hour ago|
I think a majority of content consumers can already distinguish LLM content from human content. I'm looking forward to the day that they're intelligent enough to care, but I'm not holding my breath. Orwell framed it pretty well in 1984 with the machine-generated songs that were new every year, but always tugged on the heartstrings of the proles. They weren't really readers or listeners to music or appreciators of art before, and they can be caught in the trap indefinitely, since they'll never be aware or what came before or what's being done now outside their AI-driven feed.

Horselover Fat had a pretty good take on machine generated content, too.

vintermann 1 hour ago|
It's not an "already", because I assume models will get better at addressing mode collapse.

The irony in the machine generated songs in 1984 was that Winston clearly found meaning in them, feeling like they applied to him, even though he knew they were machine generated: (from memory) "Under the shade of the chestnut tree / I sold you and you sold me / here lie they and here lie we / under the shade of the chestnut tree" - that refers to him and Julia selling each other out, right?

Just like people today - and in George Orwell's day, which was why he made it - find meaning in things which is obviously formulaic manufacured corporate slop, like the endless MCU films.

noduerme 1 hour ago||
I'm not sure the Chestnut Tree song was machine generated. [edit: I also recall Winston thinking that the proles songs were sappy and repetitive]. I took that as an older song predating the machine slop. But maybe you're right, and if so it's a sadder and deeper irony.

Finding meaning in slop is not ennobling of the human spirit, and I see no reason to champion it.

Also if the meaning is that I sold you and you sold me; what is the upside here?

More comments...