We already have the solution. Use AI to validate AI agent commands.

The future of software is fixing bugs and security issues in production.

Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.

So humans scored 66% on human eval?

Proof that people just do not read what they are seeing on their screens when put too much trust in the agent as it prints the result and they will approve anything on their machine.

So if a basic curl | bash was tweaked to download malware which the agent gets tricked into running the command but it said it was safe, the user would just approve it.

Interesting premise, but there's not much real world meaning here without stats on the percentage of agent-offered commands that are actually dangerous.

If that number is something like 10%, then we have a really big problem. But, if it's .000001%, then it's pretty vanishing. At some point in between we cross a threshold that puts the risk below many other risks that we routinely take (e.g. trusting npm dependency graphs).

Of course, if it's really that low a percentage, then the entire model of "supervising" via human approval really is fundamentally flawed.

love this so much!

[flagged]

[flagged]

[flagged]

[flagged]