Posted by ropbear 17 hours ago
At the point that you're generating entire volumes of text from Claude you're not really trying to be a sophisticated writer. I don't see how it's going to hurt for it to choose random related words.
LLMs have never been the place I've thought to expect any commitment to the craft of writing, to be fair.
Yeah, that about sums it up!
It's a good idea for many human endeavors to be able to identify AI writing. Communication, after all, is our main way of building the social fabric.
However - and crucially - good writing is still beyond the frontier of any model I've seen so far.
Watermarks for the things that truly matter may not be important at all.
Finally, as X commentators have shown, simply removing punctuation or changing a word here or adding an adverb there manually will screw up the whole process enormously.
The best will be the clever folks who retroactively apply the model distribution to fraud or other crimes to try to implicate the companies via watermark.
Gotta feel for their product, policy and legal team.
Who could have seen this coming???
"I've been working in the media and model IP space for quite many years. What this article misses a bit is the threat model for watermarking in general. Watermarking and fingerprinting have inherently weak security guarantees -- they rely a lot on security through obscurity, weak assumed adversaries to deliver. There are clear trade offs between true positives, false positives and maintaining the quality of the media. It's true for audio-visual media, models and their outputs alike. As much as I like to take shots at poor technical choices by corps and govs, this one is unjustified. Sure, inserting glyphs is bad but biased sampling is as good as it gets in 2026."
Since the announcement, there have been many people who don't seem to fully understand what a security guarantee is, what trade offs it might involve, or how popular the type of technological solution is in general (media watermarking is ubiquitous).
And naturally, there are challenges with how you will make sense of the score in your org, e.g. you wrote an email, and it's flagged as LLM-generated because you copied two generated/edited paragraphs.
Yeah, the article might disagree with watermarking as a matter of principle, comparing it to censorship. But the methodological arguments that I have read so far have been thin in these articles.
If all the providers use watermarking systems with different shifting logit weightings, and the keys are secret, you have to check every provider to see if it produced a given text.
Which is clearly ridiculous.
And if all providers collaborate and use the same weightings, or if the weightings are constant and not rotated cryptographically, a generic watermark remover becomes trivial.
That's not even getting into the legal complexities of businesses running open source models without watermarking locally.
If Gruber can't tell a fully AI-generated article from a human-written one, perhaps he shouldn't care so much.
NB: I was told yesterday it's apparently a meme to call out Claude-generated output, but here I am, as I believe it's quite relevant to the topic at hand.
If this didn't have a detectable effect on the quality of the token probability calculation, the watermark wouldn't be detectable. It may be a small degradation in the quality of the output relative to the neds of many users in many situations, but it's not zero. It's literally sometimes choosing different words that it otherwise would specifically for watermarking purposes.
If this was not the case, the encryption would be broken, and most everyone agrees that good encryption does exist.
The "quality of the probability calculations" as you put it is 100% in this case and any less would be a huge deal (as in - breaks all of the internet).
So, now you just take those same random bytes and use them as the seed for your LLM token choices. The output has the _cryptographically_ proven exact same quality as if you were using a true RNG (which you likely weren't using anyways).
You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner.
> choosing different words that it otherwise would
The "that is otherwise would" is carrying all the weight here. "Otherwise" is sampling from a distribution. You just sample from the same distribution but with a cryptographically secure, seeded RNG. https://en.wikipedia.org/wiki/Cryptographically_secure_pseud...
"Knowing the LLM distribution" seems to me like the only hard part because you don't know the context of any random snippet.
I struggle to understand the relevance of that comment.
The blue/green token list biasing process literally does cause different tokens to be occasionally chosen. Not only that, but because a different token was chosen at one point, this changes the probabilities of all subsequent tokens, and the resulting later token stream every time it happens. If you had access to the token stream as it would have been, and the watermarked one by the end of the text they will be very noticeably different.