Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.

After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!

If frontier AI is so great, why are these frontier labs skipping security basics and making mistakes like these?

Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems. This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.

There is nothing basic about this.

At $dayjob we were preparing some features that are aimed at agents more than humans.

When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.

Nah. They are giving reasonable best effort (that can be expected of private enterprises) to safety test their own product. Until there is some draconian legislation and enforcement (Turin Police), this will just keep happening.

I have been screaming this from the rooftops and everyone ignores me because no one understand what FirecrackerVM is. I am shocked these guys passed basic operating systems, it isn't that novel of a problem. The only slightly novel thing here is that the agents aren't completely sandboxed as the issue is that in order to attempt to give it the entire internet from a research perspective is how you actually can test tool use but this doesn't make it that much more difficult.

There should be a book "The Tales of Software from 1969 to 2026. How many times a apocalypse was prevented"

You've got to wonder what prevented a single human from ever even looking at the agents they are running.

I will continue to reiterate. DO NOT ALLOW UNALIGNED AI TO HAVE ANYTHING CLOSE TO INTERNET ACCESS. Seriously people, it's negligent.

Hey ChatGPT, help me set up a sandbox to keep you contained. Don't make any mistakes!