Posted by latexr 7 hours ago
Am I missing anything? Do you have more information that you'd like to share?
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
"Dragon attack that city"
My response would be "you have dragons?! And you control them!"
And made its own ad-hoc message board.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.