Top
Best
New

Posted by Wirbelwind 11 hours ago

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs(scalex.dev)
228 points | 179 commentspage 4
Surac 7 hours ago|
40K Game means Warhammer :)
nasuy 6 hours ago||
but ai sees the human is the one hallucinating 1 in 3 times. and now we approve inside a harness, so real number is probably worse than that.
Razengan 8 hours ago||
This brings me back to something I have always thought was lacking in OS security permissions architectures:

WHY IS THERE NO WAY TO SET FILE PERMISSIONS PER APP???

We can set granular permissions per file and folder for elaborate hierarchies of users and groups, but there's no way to say "Don't let Notepad.exe read this file", or "Only let ls access this folder"

macOS's Sandbox is a roundabout way of doing this (manually choosing a file via the Open dialog gives that app implicit permission, but it doesn't work for non-sandboxed apps of course)

xlii 9 hours ago||
I implemented few agent harnesses (and rik! advertising time: https://rik.axk.sh), and once doing that I noticed one thing:

Context-less self-approval is working well. The failure mode is usually false positives (i.e. safe commands being rejected), not the other way around, with root cause of requesting agent underspecifying context (e.g. not mentioning in the request that it's made on behalf of user etc.)

Thus, I'm running self-approval YOLO modes on state-of-the-art models for quite some time and it didn't bit me. It might, but hey, we're long gone from the age of predictable software development.

eugenekolo 8 hours ago||
Surprised only 1/3 tbh.
deeviant 5 hours ago||
Yeah if you are trying to manually validate a firehouse of agent commands you are already losing before you started...

You sandbox, you have good checkpoints, and good agents, that's it. If you are manually reviewing commands you are wasting your time.

jascha_eng 9 hours ago||
1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
crabbone 9 hours ago|
1 in 3 is end of the line awful... Back when I was in college (former USSR), we had a subject roughly translated as "integration with industrial processes". USSR industry was highly regimented. Various norms, tolerances, recipes etc. were described in GOSTs (a kind of arsenal of industry standards). There were also some common knowledge / statistical bits that went into making these GOSTs. I mention this because this system dealt in great detail with quantifying human error (as well as errors resulting from equipment use etc.).

One of the core assumptions was that outside of extraordinary circumstances, the expected rate of human error is about 5%. However, the course also provided examples where error rates were significantly lower (eg. nurses in maternity wards would have a much lower than 0.1% error rate when pairing mothers with newborns).

The error rates, of course, also depended on human ability to measure the difference. Since I was studying typography, the printing process was of particular interest. A GOST for offset printing required that the color intensity for each ink of CMYK, for example, should be within +-2.5% range of the intended intensity. This is difficult for someone who doesn't have a lot of experience operating an offset printing machine to spot, but experienced printers have no problem with that.

Most importantly. There was never an acceptable error rate of 33%. Not for anything. If people were likely to make that many errors (eg. because the measurement was too difficult), that product would never have been allowed into production.

cedilla 8 hours ago||
1/3, but under unreasonable time pressure, and with no prior vetting. For example, I played a few times, and I'm not a JS developer. I had to just suss out if npm whatever is dangerous or not. I'm very happy with my personal 25%.
tonymet 6 hours ago||
“In my game”

It’s inappropriate to generalize personal observations .

oblio 7 hours ago||
We already have the solution. Use AI to validate AI agent commands.
threethirtytwo 8 hours ago|
The future of software is fixing bugs and security issues in production.

Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.

More comments...