Posted by ropbear 15 hours ago
But even if there were such a thing as a truly "best word", for some context, what are the examples here? Mango vs pineapple? Gray vs overcast? In what case is one of these better, that AI would normally infer but would suddenly be "perverted" by SynthID? Do you think your emotional state and preferences are being evaluated if they aren't explicitly in memory? And if they are there, do you think that the generator will bypass those instructions in favor of the watermark instead of placing it somewhere you won't care? I just. Genuinely don't get it. There may be words that matter in specific contexts or to you as a reader, so you should bloody well put them there.
Has that ever been the case? Are they not actively tweaking their models, their fine tuning, the system prompts, the tool definitions and implementations, the guard rails, tool calls, instant responses. There are hundreds of knobs that they can change daily, or between each prompt, or even half way through a generation.
Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?
This entire article just seems so detached from the basics of how LLMs work.
No and no. I am not sure I agree with his point but I know he is not ill-informed on either of these points, because I mentioned them to him a couple of days ago.
The point I made (quite briefly) is that watermarking is only feasible because for good writing it is necessary to use T>0, or the writing will never explore a more creative choice, and that at T=0 you don’t even need a watermark to spot LLM-generated text.
The point he is making is consistent with this, isn’t it? Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure. These are ethically distinct approaches, and since he disagrees with the EU objective he comes down on one side I guess.
Me, I don’t care about the hypothetical enough.
Not least because I think Claude writes depressingly badly and I doubt any steganographic change will enrage me less.
He repeatedly states that choosing "the best word" is the most important thing to him. I don't know how you reconcile that with creativity itself, let alone probabilistic sampling.
That doesn't actually give you the 'best' text in any human sense of the word. Just like playing the 'best' move in Poker without sampling leads you to lose a lot of money.
This framing does not make sense to me. What do you mean by "influenced and analysed"? How have you or anyone been influencing or analyzing the randomness behind the sampling process to create better writing? What makes the unadulterated randomness "driving creativity" but a different random choice uncreative?
You're mischaracterising or misunderstanding my point, or I mangled it.
I mean it is possible to analyse, control, monitor, study the impact of changing temperature on the writing, yes?
The point about watermarking is that this relationship — change the temperature, see the effect — is now being adjusted by an unstated, secret process you explicitly can't control.
(I gather Anthropic have recently taken away this setting anyway; that was news to me.)
You are projecting that onto me, and I cannot tell you how comically poorly aimed it is.
His first take on this situation was cutely naive, thinking they were going to inject secret hidden unicode characters. But ultimately he has a massive hate on for the EU -- they were mean to Apple once -- and it comes out in any topic that overlaps.
That's ... even worse? So we're all here in the comments trying to figure out what the author means, and what their overall point is, while clearly they don't even use the damn thing? Oof... What a waste of time for everyone involved.
It seems fully logical to me that someone who writes for a living (who, as it happens, developed the very markup language LLMs use for everything) should be invested in understanding the automatic plagiarism and word calculating machine from an intellectually honest position.
I personally am pretty severely big-two-AI-firms, increasingly anti-big-tech, but I am learning and researching uses of LLMs because for myself I really need to understand how to use them in an intellectually and (as far as is possible) ethically sound way. Learning because as a boring old freelance programmer I have to; foolish to pretend otherwise.
So I completely understand his position — that the AI industry is hot air and crooked and scammy and weird, and some of the people involved genuinely rather dark-sided, but the technology exists and if it hints at threatening your livelihood, you need to understand it.
From reading his work for the best part of twenty years or so (and emailing him intermittently over that time) it would seem to me that he's a lot less bearish on the tech industry than me, and a lot less fond of the EU than I am; he's more optimistic than I am. But he writes because he has to write. I should think that would make him highly invested in understanding what LLMs do.
Seems like he really likes to use LLMs and is worried that quality will be degraded. But he will never demonstrate such degradation scientifically, we don’t have anecdotes even.
I think that was intended, yes.
It's still possible to use Claude to proofread - highlight grammatical, flow, structure, logic errors and make simple suggestions for you to pick and choose or adapt as you wish. No watermarking will flag your text. No flaw accusations of LLM authorship will haunt you. All will be fine.
But if you want an LLM to rewrite your text, that's (a) not proofreading, and (b) should be flagged as LLM generated ... because it is.
You could use your own hypothetical house elf to do it for you, or pay someone to do it. LLMs are just cheaper for a certain set of problems.
People will find ways to circumvent this, so this limitation will only hit the technically less adept people.
In the fact that you didn't write it.
> You could use your own hypothetical house elf to do it for you, or pay someone to do it.
Yes, and those would be similarly problematic (and more expensive).
This is a fact and this is generally not a problem. Customer support guy Joe did not write that email to you with a refund: someone else did it and Joe did pick the template. Alice did not write that post card to Bob, someone else did and she just googled some nice text. We deal with a lot of content that wasn’t written by the person who signed it. That content, when written by LLM, may indeed contain watermarks and nobody will care about the choice of words, because only the meaning matters in such communications.
People pay too much attention to authenticity here, which is no more than a demonstration of an effort. LLM can and should write scientific articles because the real effort is in directing research, not summarizing it. LLMs can and should write news, because it is cheap and efficient, and real reporting is in discovery. LLM can and should write fiction and make movies, because there is no reason why creators of various junk should earn their money easily. LLMs do not replace real talent. They just emphasize for an average person how easily replaceable they are. And that‘s ok. Creative industry is a blue collar job now.
Factory farming also makes meat cheaper than organic practices. I'd just like to know which one I'm getting.
And if you copy-paste the answers from LLM, I think it's only fair the end result gets flagged. You're not writing it yourself.
I wonder if it would even get flagged in that case, because wouldn't the probability distribution of a token when the LLM is suggesting an edit to your writing be different than the distribution of that token once it is in the context of the text it's editing?
They will need to dodge around the EU requirements but it will probably just come down to an alternative method to watermark or a contractual assurance you won't mis-represent the source of the text.
I hope the other providers will add a geographical limitation on this EU rule.
(On a side note, I wish they would replace those EU beauracts with LLms).
but proof reading is a linter, not a writer. the proof reader will say "I think this is clumsy can you try x,y & z"
A point of confusion for me, however: is every watermark unique? Is every algorithm for watermarking going to vary amongst models and amongst model versions? Will each model publisher keep this watermarking as a trade secret, that they alone can detect? If so, this can't scale! How do you detect "JoeBob 4.3 LLM" output? By querying every single model's watermark-detector? And if they all work by re-running the model and using tokens anew? That is extraordinarily wasteful.
If a watermark is not self-evident, or universally detectable, then it is no good. Take, for example, US currency. The security measures are published and well known. Any count-out room in retail has a big poster indicating how you can detect authentic US bills. Nobody has to accept non-US currency in the US, and so the only authenticity you need to worry about is your US bills alone. LLM watermarking has none of this in common. Currently sounding like a shitshow, if you ask me.
Wouldn’t having that be enough to eventually reverse engineer the key?
Removal may come down to changing every third token to a different one.
Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now?
If a human proves human authorship, and LLM watermarking tests positive, then is that going to be considered a "derivative work" or not? What if there is an applicable license for the source work, such as "CC-BY-ND" that prohibits derivative works?
This has not been court-tested, and I expect that it will need testing at that level before we can have any assurances.
As for it being "uncopyrightable" if it were the output of an LLM, I think computer people are making a very aspie interpretation of a single decision. I think it's more that the LLM (and thereby its owners) cannot itself hold a copyright on its output, that output has to be touched by a person before it is copyrightable. A particular view from the top of a mountain can't be copyrighted, for example, but a photograph of that view can be.
I'm not sure it at all precludes a "robosigning"* sort of situation, where machines generate output, hired temps sign and claim that output, and immediately sign it over to the people who hired them (as a work-for-hire.) Copyright is stupid, artificial law, not logical.
-----
* https://www.mortgageauditsonline.com/what-are-robo-signers/
For the watermark to be detectable, the text needs to be like 75% AI generated.
If you have an LLM “touch” one section of the article, it’s not gonna be detectable.
Then why are you using an LLM to write? They're not capable of understanding such nuance. They do pick randomly between two synonymous phrases, they do not use some super smart algorithm to pick the one that sounds the best.
This excuse doesn't hold any water at all - Occam's razor says the author is just super annoyed that his AI writing will be identifiable as AI writing.
Alternatively, you ask Claude to write the whole thing and proof read it yourself. In that case I'd like to know how much you'd need to change to break the watermarking, i.e. how much of a text would you need to change for it to be considered your work and not that of Claude?