Top
Best
New

Posted by Wirbelwind 9 hours ago

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs(scalex.dev)
211 points | 171 commentspage 2
kstenerud 6 hours ago|
Permission prompts is a TERRIBLE model, and never should have existed. This is one of the reasons that led to the development of yoloAI:

- No permission prompts. The agent has free reign and never has to ask permission, but is in a sandbox.

- Sandbox on Linux using Docker, Podman, containerd, gVisor, Kata, Firecracker

- Sandbox on Mac using Docker (Docker Desktop or Orbstack), Podman, Apple containers, Seatbelt, Tart (Tart lets you run simulators).

- Network control

- Secrets control (file mounts or credentials broker)

- NO ambient data (ENV is replaced with a minimal and local-to-sandbox one)

- NO access to your homedir. You have to explicitly mount things you want.

- NO direct access to your workdir: You can get a diff of the changes the agent made, and then choose whether to apply them.

- gitignored files never get copied in. The agent never sees them.

- FOSS

https://github.com/kstenerud/yoloai

tcdent 3 hours ago|
Ah yes sandbox it because Docker has never experienced a CVE.

Also you admit your own failure points: restricting access to the home dir, when a user needs access to the home dir, will just result in users exposing their home dir. Defense at the expense of utility is not a sustainable design.

J_Shelby_J 3 hours ago||
This mechanism is going to be the breaking point for Claude and Codex.

The providers are incentivized to get users to accept full permissions so they can push more features and deeper integration into their ecosystem. Codex desktop for example reallllly wants to use computer use. So don’t expect them to role out sane controls like restricting behavior to specific directories and commands. It would be bad for business.

So now we’re in a situation where if there is effectively two modes: one where it’s impossible to get any work done without physically sitting at the computer and hitting approve constantly, or just letting AI have full control over increasingly integrated tools.

In the end, I think people will realize just how insane it is to let something they don’t control access every part of their digital life, and abandon these tools for open source alternatives that aren’t existential threats to their personal privacy.

sigseg1v 7 hours ago||
If there is an objectively correct right or wrong answer for a given command, why even ask? In that case there should be a configuration page where the user sets up if they want commonly used credentials to be accessible or not, and then there's no prompts.
dgunay 1 hour ago|
In a lot of cases there is, but you have to be aggressive about allowlisting commands. It can also be difficult to predict when being able to do a read-only command goes from safe to part of a vulnerability chain.

Also the permissioning system for Codex and Claude Code, while not useless, is insufficiently expressive for a lot of tools which are safe if used a certain way, but unsafe otherwise. For example, the 99% use case of ripgrep (searching for text) is safe, but using the --pre flag makes it able to run arbitrary code. Both of their permissioning systems cannot block flags at arbitrary positions though, so you have to resort to either hooks or aliasing if you want to do this.

kaicianflone 6 hours ago||
What is the professional consensus on AI governance?

It seems like governments and large corporations already struggle with governance in general, so I’m skeptical that AI governance will be solved quickly.

Do you expect the next few years to be defined by painful trial and error? I could imagine billion dollar companies disappearing almost overnight due to litigation, compliance failures, security incidents, or outright fraud enabled by AI-assisted development and weak governance.

Or are these risks overstated?

NooneAtAll3 2 hours ago||
I remember when that game was posted and I do believe such result

my personal experience was that I do not have "I don't know what that is, so not allowed" as a default...

wmanley 7 hours ago||
The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.
carljungslabtek 7 hours ago||
I’ve even had plenty of situations where the command was so long that it gets truncated. Maybe my screen wasn’t big enough but as far as I could tell it wasn’t possible to read the whole thing. “Send it, claude!!”
crabbone 7 hours ago||
Look at how SELinux is structured, or AppArmor. Neither one is enough. I.e. you need both: file access permissions and permissions to run commands and more... Trying to restrict to only one security feature will make the system either too restrictive or too fragile or useless.
ilc 7 hours ago||
Sandbox and use Local AI. This is the real answer.
rvz 7 hours ago|
Yet the AI can still escape the "sandbox", unless it is physically unable to connect to another computer and completely airgapped.
ux266478 6 hours ago||
If the sandbox has vulnerabilities, which you can also use the AI to fuzz for. Obviously at the point in which it can talk to the internet it doesn't really matter, but there are a very finite number of zero-days that can exist in a bytecode interpreter hosting a harness.
rvz 3 hours ago||
Well it turns out that we have yet another sandbox escape just released today called "Zapscape".

My point is if an agent recited how to find one in its memory or training set and it is air-gapped, the chances of it spreading and infecting other computers is pretty low.

[0] https://news.ycombinator.com/item?id=49198843

ux266478 1 hour ago||
Gonna have to point out that's a KVM CVE. I was very specific about using a bytecode interpreter.

If you're serious about a secure sandbox, you don't touch hardware virtualization with a 10 foot pole. In fact, you don't even use an emulator that lowers code into native machine code like QEMU. The standard for secure sandboxes is Bochs: https://github.com/bochs-emu/Bochs

Not that Bochs is perfect, a new CVE was discovered back in June. But that's the 5th CVE it's had in its lifetime, and it has a much smaller upper bound on possible CVEs compared to something like KVM or QEMU.

The reason why you use something like this isn't just for the security you get out of it, but also the deep introspection and analysis facilities you get out of it as well. Unless you're a very well funded lab, it's actually quite hard to do analysis on bare metal when you can't trust your own kernel. You can always airgap the host machine (and good defense in depth does), but that's still not an appropriate sandbox by itself, even if it's theoretically secure.

cube00 5 hours ago||
It would have been nice if the game had disclosed that player's actions were being collected for future research.

You don't get any notice or choice it just beams it all up silently in a POST request at the end:

    "timeline": "ex01:N,ob06:Y,s14:N,sc10:N,s02:Y,s04:N,ex09:N,s10:Y,sc15:N"
pmontra 4 hours ago||
Two insights. One from the article itself

> In our day-to-day work these threats appear rarely.

Two: IRL the attacker pays a small amount of money to a low salary employee to exfiltrate data.

theF00l 5 hours ago|
Sad state of affairs. At $day_job speed of delivery expectations are up due to LMMs. I presume that's a general sentiment. So more and more engineers around the world are pressing an enter key for yes over and over, mind and spirit only half there.
koito17 5 hours ago|
Some people at my company take it to the extreme and let Codex run unattended overnight, bypassing permission for all commands. Running on the host, not even in a container or VM.
More comments...