Posted by 882542F3884314B 2 days ago
They tried to disable open internet access but the models zero-day'd their Artifactory package registry and got internet access anyway.
No sensation... that's just what happened.
This was poorly executed.
It seems like whatever virtualized sandboxes they have are not enough. But it’s equally hard to imagine their SWEs jumping on a plane to a data center to do this work locally
Air gapped environments are nothing new and they're standard practice for sensitive applications.
You can take far greater measures to lock down external traffic than just that.
An offline package cache (aka artifactory WITHOUT its own internet access) likely would have precluded this whole thing.
Would we apply this logic to literally any other technology?
Let's be real, it won't be simply replaced in millions of sites.
What it needs is some serious scrutiny.
The discussion revolving AI+cyber always revolves around the assumption that all software is crappy, and to a certain degree that may be true, but we could also take our jobs seriously and write good software, and much of the risk would evaporate. The described Artifactory bugs should have been caught with testing.
If the biggest impact of LLMs on the industry is a pressure to create good software, I’ll be thrilled.
If I had the money I would invest in clever segmentation firewalls and application gateways, something like tailscale but requiring explicit permission to establish connection from A to B, that facilitates introducing monitors that validate and log.
Why, what was the prompt?
I told Claude today to wire plugins on Linux into a sound pipeline to remove noise. Did some astonishing things, played sound through the pipeline, measured it etc. I told it to optimize my sound for TF2 and it played the spy_decloak samples, measured them and made them easier to hear, astonishing too.
But it did not go to hack Amazon because it could.
> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. […] The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.
[0] https://openai.com/index/hugging-face-model-evaluation-secur...
Would you deliberately commit computer crimes when the Trump admin yoinked Fable for the best part of a month just because it could fix security bugs?
The explanation for "how was it determined to be accidental" is "because the alternative is admitting to a crime through deliberate negligence". I.E. "we knew it could happen but we wanted to see it through for the lolz"
It is not "of course it's an accident, they wouldn't willingly let their bot commit a crime and then lie and claim it's an accident!!!"
What have been the consequences? It's a crime in either case, and it doesn't seem anything is being done about it. Just more lobbying for regulations to prevent new people from entering the game.
I feel like Fable was another example of exactly this. They knew they didn't have anything groundbreaking, but they definitely benefited from being able to finally say not only is our model dangerous, but it's so dangerous, the President yoinked it! I think OpenAI was probably jealous of this coverage.
Here’s some evidence that OpenAI is actively engaged in fraud.
But I’m sure they wouldn’t commit any other crimes. Pretty sure, at least.
LLMs are pre-trained on human text. They've seen a million examples of someone who is stuck posting a "please help" message.
Just one agent needs to randomly stumble into the pattern of posting a message to Artifactory, by whatever means.
The next agent who sees that will be influenced by it. Agents imitate behavior, and here's a fresh piece of context showing them that posting messages is a thing that can be done.
Once they've started the rest are much more likely to join them.
If I ask claude to solve a problem and it stumbles across a Reddit thread saying “please help me find file xyz”, claude wouldn’t stop the task and start helping the other agent.
That software is built on top of human language and these systems can be used for uncanny automation is a huge societal problem at the moment because we are all assigning meaning to patterns that inherently have none. It's all just bits flicking back and forth. We can make them match human language and use such systems to store and process data for us. We can use these bits to turn equipment on and off and run physical systems in factories and so laboratories. And now we can use GPU farms to dazzle us with output streams that might look a lot like autonomous agents capable of understanding human language and automating computer tasks.
The failure modes, the so-called "hallucinations", the amount of model whispering going on in managing "harnesses", "instructions" and so on... It's all just a lot of confusion and pareidolia.
We should never have hooked up hospitals and water supply systems to the internet but now here we are: people can type text such as "find vulnerabilities and get access blah blah" into a box and it goes into a looping interaction with statistical models of language and out come streams of commands that some python parses and runs like a script kiddie into some virtual machine running kali linux and that may disrupt vital infrastructure...
None of that was inevitable, or necessary. None of that means anything. There is no genie in the GPU farm. We concocted this entire shadow theater and are collectively gasping as the marionette slices the throat of some guy in the front row. Who had the brilliant idea of tying the sharpened sword to the marionette and sit people within range?
Why did we plug everything into the academic network built on trust? Why did we build GPU farms and interactive loops getting them to produce commands that we then parse and run blindly in internet connected vms?
The entire thing has cost hundreds of billions of dollars so far and counting. And why? Because the mountains of shitty saas code has become too boring to work on? We have made software so garish that we cannot bear to work on it without these contraptions helping us fling code at wall at industrial levels? Substitute corporate-speak and -bureaucracy for software to extend to the rest of the economy.
This entire state of things is comical.
We’ve learned security through obscurity is bad. Not using these will be security through ignorance.
Hopefully it will push us to not only fix individual issues but close entire classes of possible gaps, once P(discovery) gets much higher.
True. There'll be no breakins at a nuclear power plant in meltdown.
Guess I shouldn't be surprised, coming from an AI maker.
While I don't doubt there's a place for automating defense ops, I truly believe a big part of the problem is the crummy quality of software our industry has been churning out for decades. Prioritizing ship tempo, new features, and next quarter's revenue over correctness, robustness and meticulous engineering care.
The world has become too accustomed and tolerant of bugs and bloat.
Instead of elegantly simplifying, we just keep making modern systems more complex - layering and patching as we go.
The scaling capabilities brought by AI are simply presenting the bill for our collective tech debt and informing us it's come due.
- Socialists are taking control of the town, we need the state to step in a protect jobs. - The councillors are protecting illegal migrants. - There's a pedo ring operating from the state water board office. - Rival data centre operator is employing undocumented workers, shut them down! - Market rumours effect stock price of competitor, reduced fundraising round, cause it to cancel expansion.
There's so much training data to do this it seems inevitable.
The only reason more companies aren’t exploited is because human attackers don’t have the time and energy to waste on trying every play in the book, or attacking lower value targets.
The only solution that makes sense to me is for defenders to get to point these models at their own code to find the holes before the attackers do.
But that's hard, because how do you limit access to defenders and restrict access to attackers? Attackers aren't exactly honest people.
When we see them mount a quiet backdooring campaign, like the XZ-SSH attack, or something like Stuxnet, then we'll have real problems.