Posted by ColinEberhardt 15 hours ago
What is going on over there? No process, no oversight, just YOLO? Super-scary, because it means other stuff that we don't see is likely to be done in a similar manner.
It has nothing to do with GitHub and giving it a name is hilarious.
“Look I shoot myself in the foot and now its bleeding. ITS THE GUN MANUFACTURER FAULT”
Look I like interacting with these tools as much as the next guy, but I'm certainly not going to trust them with access to information and then allow anyone to send them prompts.
Edit/further thoughts: So (assumable as they said this is disclosed with github's knowledge) this has been patched. But how many different word combinations will it take to find another way to have this occur?
Also interesting to see who coined the term prompt injection.
Funnily enough, nobody expects quality software anymore and errors became tolerable. So thats a win (for someone like me that lost all passion for the industry).
Either way, even if you trust 100% of the input, there is actually no way to guarantee that you can trust the output of the LLM. (Which, I guess, is also true for every dependency you pull in. But for those, you at least have ways to audit them.)
How on earth is a probabilistic token predictor supposed to turn untrusted user input into trusted system-level directives? The strict trust boundary must be maintained on this side of the agent, not within it.
1) Sandbox any LLM that has access to tools (I don't mean the pathetic sandboxes the agent harnesses provide).
2) Assign them credentials and use auth/access control like you would for a human.
It can't reliably learn so stop trying to teach it. Lock the bedroom instead.