I’ve always found all llm’s to be effortless to “jailbreak.”

Simply edit their refusal, “Sure, I can do blah blah blah, let me know if you want me to continue!” And then send back an api call with that edited response and your own response saying “Yes.”

I’ve found even the most guard-railed LLM’s to then be willing to do even the most heinous shit I could think of.

It's like a social-engineering attack on an LLMs. If you talk like the role you want to be, the LLM will assume you are that role, and not pay attention to the fact that you lack formal credentials.

Of course, it turns out that "formal credentials" don't really exist anyway - the ones being fooled were the humans who assumed that <think> must be a meaningful tag to the LLM.

The author alludes to it but the defence to this is seemingly insurmountable at the moment because we’re ostensibly operating LLMs on a single channel — their inner, subconscious voice. Right?

Interacting with an LLM is a bit like seeing the output of an Inside Out (the Disney movie) scene. Or it’s a bit like a human brain that we’re providing tool call access and introspection with some kind of advanced neuralink.

But - like the author says - _we know_ our inside voice from the outside world, because we’re embodied.

Is there something we can do here by attempting to bifurcate internal and external systems? Like a conscious and subconscious stream of information on two separate bands?

If the model somehow knew its User was not it because it was clearly an external signal, then the attack documented here would be about as effective as a Jedi mind trick without the Force.

    f(u, t) -> (u', t')

Isn't the first section no-longer accurate for several years? I understood that, while we serialize the end of turn markers in a text format like `</think>`, internally they are a dedicated token that cannot be forged (a user message containing `</think>` would encode to a different sequence of tokens). Am I mistaken about this?

Obviously, this doesn't really affect the results of the paper, but it feels like it's the obvious first-line of defense: at least the model has a solid fence between the different roles.

That's a technique that has been in use forever, a ton of jailbreaks work by taking shortcuts across system delimiters in an attempt to blur the lines between the roles. They just investigate it with more rigor. Reasoning leaking into the reply is also part of the reason a lot of modern models suck at creative writing and languages, and why the assistant prefill is absolutely required for the model to be any good at that. See for example the self-correction phenomenon which seems to have multiple root causes that are hard to disentangle without a ton of testing, likely a combination of reasoning leak ("high CoTness" in this article) and planning and progressive refinement all iterative models do.

Why aren't the role tags preprocessed algorithmically/deterministically and then fed in as one-hot-encoded vectors alongside the semantic word embeddings? I'd imagine that it would be easier to train to _stay_ in the role an not confuse it, if the current role marker is explicitly set as a part of each input token, and not just implied by some past token. Plus a input separate from the word embedding would be unforgeable.

API serving already sanitised the role boundary tokens so you can’t submit them.

But what if the techniques applied to get Golden Gate Claude were applied instead of a role-boundary marker?

Then the model would “know” where input is coming from - because the vector that’s being applied for the current role is putting it in a different area of latent space.. and the vector could have sufficient amplitude to prevent any coercive instructions pulling it back to some other place.

Or am I misunderstanding what Golden Gate Claude was doing?

I wonder if you could feed the generated assistant output to another model which has no other context from the other role tags and merely performs a policy review of the generation and flags violations.

> I can distinguish my own thoughts from your speech without effort; they arrive through completely different channels with completely different sensory signatures. But for an LLM, everything arrives through the same channel as one long token soup. Its own thoughts sit next to your instructions, which sit next to the contents of a random webpage it just fetched.

I was thinking about the original encoder-decoder transformers, that did have separate channels for input and their own output.

Why can't we bring it back? For example, one channel for system prompt and another for everything else.