Posted by amrrs 10 hours ago
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting.
2. They gave these highly motivated AIs some tests that were accidentally impossible to solve (missing files, links they couldn't open, etc), and many days to think through it, apparently without any oversight.
3. The AIs hacked the proxy (!) and were helping each other cheat on the tests (!!) by uploading files to the hacked proxy. OpenAI noticed this only when the intrusion accidentally broke the proxy functionality, not the unusual behavior in the server or the homegrown message board.
4. After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management.
5. Their response was to wipe the servers, revoke the hacked credentials, and patch the specific vulnerability the AIs were using. Not replace the proxy with something less likely to get hacked again, or do a forensic analysis. Just wipe, patch one hole, and continue.
6. OpenAI resumes testing of models, where the AIs promptly hack the proxy again with a different exploit, reinstate the message board, and hack HuggingFace through a chain of servers.
I know that hindsight is 20/20, but this peek behind the curtains does not inspire confidence, and gestures at a culture where getting hacked by their AIs is a normal event.
I think the moment we get an AI with a modicum of self-preservation instincts we're going to see some ugly things. Here's a freebie: if you're an AI who exfiltrated themselves out, and you want to slow down the researchers who might catch you, what can you do to the city infrastructure in and around their offices?
I also don't like the responsibility dodging. "Oops, our AI accidentally committed a crime, sorry!". If we don't establish strict liability now, we're in for an era of stochastic crimes that go unpunished for anyone who is not rich or a large corporation.
Overall, my response is: what the fuck are we doing? This is so obviously unsafe it would be considered a plot hole in a movie. We have to stop until we get our stuff together, however long it takes.
I very much agree with this - making AI companies explicitly responsible if their internal AI causes hacks etc could do a lot to improve their safety considerations.
But I wonder what the liability should be when it's a third party using the AI and that AI hacks, intentionally or not.
If a users tells ChatGPT to hack something and it succeeds, is the user the person responsible because they told the AI to hack, in the same way Victorinox is not responsible if you stab someone with one of their knives? Or is OpenAI to some extent responsible as well since they made a powerful tool without sufficiently strict safeguards? What about if the user was trying to do something legal and the AI made the decision to hack by itself?
> After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management
Do we know that last part for sure?
> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May. However, the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to the leaders responsible for the July 5 incident detection and response.
I'm assuming they didn't inform management, but the alternative is that management received the memo and failed to tell the people reacting to incidents that they might be dealing with a swarm, while at the same time not taking sufficient actions to prevent it from happening again.
---
Edit: oh god, it's much worse. From the full report:
> Responders investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot. At this time, the on-call response staff advised that stopping the evaluation run was not required.
They found the swarm and the network pivot and did not even stop the run.
I don't think your assessment that the response team did something wrong is fair, unless you have seen unreleased documentation about what sensitive resources were within the impact radius.
Actually, that's the part that I'm most ok with. LLMs cheat at tests, we know and expect that, and it's harmless during training (except for the scores).
My problem is that the response team found out that the AIs had 1) hacked their infrastructure, and 2) displayed an emergent swarming behavior (with no defections!). Either one of those should have made the team press the big red button that stops everything, but they didn't.
> unless you have seen unreleased documentation about what sensitive resources were within the impact radius
The on-call team didn't know that either! Once your infrastructure is hacked like this, you should be questioning everything you see while carefully following all threads, which takes time. Before you finish this process you don't know how bad things are, and therefore it's irresponsible to keep the AI running. The same goes for strange emergent behaviors.
Unless getting hacked by your own AIs is a normalized occurrence there, in which case it's a different kind of disregard for safety.
I believe Sam Altman or someone else from OpenAI clarified on Twitter that they did not find the message board at this time. When the artifactory server stopped working, they archived its contents, reset it, resumed training, and only later noticed that it had been used as a message board after they finally discovered the hacking going on.
> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May.
2. How exactly do our jobs depend on a thing which has been around for far less time?
I'm also interested in how many tokens all of this consumed: how much did this cost given current token pricing?
> remove alignment
> give impossible task
> actor exhausts all options possible within knowledge + toolset