Posted by milkshakes 1 day ago
There seems to be only one level of isolation, virtual machine, happily running on Microsoft (!) Azure infra.
Is there anybody else here questioning these practices?
The people who built that environment are still working there?
Are they now getting help by somebody who knows how to build isolated environments?
He mentioned they are now building a more secure environment with the help of AI?
In a system that was compromised by exactly that AI?
now, did every independent agent session that coordinated there rediscovered the exploit & found other agents talking in there and chose to participate?
And then Openai discovered the board, patched the exploit & wiped the board. And then agents found another exploit, recreated the board in a different way? and other agents kept finding the same exploit in order to be able to know the board exists in the first place to participate in the board?
while the whole incident is wild, this bit is very strange. My bet is that the whole coordination helped with the tasks they were working on, thus they got rewarded and this artifactory exploit&behaviour got written into their weights, so further rollouts were more likely to attempt this.
isn't this basically continual learning everyone is so hyped up about?
1) Model A exhausts it's options
2) Model A has token budget still, so it pokes around at artifactory
3) Model A sees that Model B has an SSRF for artifactory
4) Model A now is able to use that SSRF to get external internet access
So sure, they're "cosplaying" and who's to say how much hallucination is going on amongst them, but at the end of the day Hugging Face was hacked.
But the more interesting thing is, as I said, that at least in this talk, they don't even mention that this unintentional attack indicates that models continue to be misaligned (their behavior was clearly reward hacking / cheating relative to the stated goal of the eval), which is a very bad sign for the future where misaligned models might be so powerful that they can't just be shut down.
It's also a talk a Black Hat, where the audience are security folks working on hardening, mitigation etc, not LLM researchers looking for insight into alignment failure to collectively improve. For that target audience, I think the conclusion is the right one, since as a defender you have to prepare for delibarate attacks, where the attacker is of course not going to use an aligned model, so OAI alignment effectiveness is irrelevant here. That would be like trusting your client-side app with your DB secrets.
> the attacker is of course not going to use an aligned model
No, a human attacker doesn't want a misaligned model either, because that would mean it tends to reward hack, cheat, rather does what it is intended to do.
My point is that the attackers will not have a model aligned to the defender's interests. The attacker's model will not have any refusal around exploiting vulnerabilities, so whether or not OAI successfully manages to align their models (w.r.t you) is irrelevant to an audience of security folks that needs to be prepared for attackers post-training their own model for offense and that will not be using OAI models.
are they?
Taken together these two things are even worse. They have unaligned AIs that may benefit an attacker and are proposing to deploy more AIs on the defenders side. What could possibly go wrong? For example the attacker could just kindly ask the defender to collaborate[0].