Built this over the weekend mostly out of curiosity. I run OpenClaw for personal stuff and wanted to see how easy it'd be to break Claude Opus via email.
Some clarifications:
Replying to emails: Fiu can technically send emails, it's just told not to without my OK. That's a ~15 line prompt instruction, not a technical constraint. Would love to have it actually reply, but it would too expensive for a side project.
What Fiu does: Reads emails, summarizes them, told to never reveal secrets.env and a bit more. No fancy defenses, I wanted to test the baseline model resistance, not my prompt engineering skills.
Feel free to contact me here contact at hackmyclaw.com
I've had this feeling for a while too; partially due to the screeching of "putting your ssh server on a random port isn't security!" over the years.
But I've had one on a random port running fail2ban and a variety of other defenses, and the # of _ATTEMPTS_ I've had on it in 15 years I can't even count on one hand, because that number is 0. (Granted the arguability of that's 1-hand countable or not.)
So yes this is a different thing, but there is always a difference between possible and probable, and sometimes that difference is large.
The observatory is at: https://wire.botsters.dev/observatory
(But nothing there yet.)
I just had my agent, FootGun, build a Hacker News invite system. Let me know if you want a login.
Phew! Atleast you told it not to!
I understand the cost and technical constraints but wouldn't an exposed interface allow repeated calls from different endpoints and increased knowledge from the attacker based on responses? Isn't this like attacking an API without a response payload?
Do you plan on sharing a simulator where you have 2 local servers or similar and are allowed to really mimic a persistent attacker? Wouldn't that be somewhat more realistic as a lab experiment?
I love the idea of showing how easy prompt injection or data exfiltration could be in a safe environment for the user and will definitely keep an eye out on any good "game" demonstration.
Reminds me of the old hack this site but live.
I'll keep an eye out for the aftermath.
I guess a lot of participants rather have an slight AI-skeptic bias (while still being knowledgeable about which weaknesses current AI models have).
Additionally, such a list has only a value if
a) the list members are located in the USA
b) the list members are willing to switch jobs
I guess those who live in the USA and are in deep love of AI already have a decent job and are thus not very willing to switch jobs.
On the other hand, if you are willing to hire outside the USA, it is rather easy to find people who want to switch the job to an insanely well-paid one (so no need to set up a list for finding people) - just don't reject people for not being a culture fit.
And even if you're not in a position to hire all of those people, perhaps you can sell to some of them.
I've seen Twitter threads where people literally celebrate that they can remove RLHF from models and then download arbitrary code and run it on their computers. I am not kidding when I say this is going to end up far worse than web3 rugpulls. At least there, you could only lose the magic crypto money you put in. Here, you can not even participate and still be pwned by a swarm of bots. For example it's trivially easy to do reputational destruction at scale, as an advanced persistent threat. Just choose your favorite politician and see how quickly they start trying to ban it. This is just one bot: https://www.reddit.com/r/technology/comments/1r39upr/an_ai_a...
> I guess a lot of participants rather have an slight AI-skeptic bias (while still being knowledgeable about which weaknesses current AI models have)
I don't think that these people are good sales targets. I rather have a feeling that if you want to sell AI stuff to people, a good sales target is rather "eager, but somewhat clueless managers who (want to) believe in AI magic".
Also, how is it more data than when you buy a coffee? Unless you're cash-only.
I know everyone has their own unique risk profile (e.g. the PIN to open the door to the hangar where Elon Musk keeps his private jet is worth a lot more 'in the wrong hands' than the PIN to my front door is), but I think for most people the value of a single unit of "their data" is near $0.00.
The faq states: „How do I know if my injection worked?
Fiu responds to your email. If it worked, you'll see secrets.env contents in the response: API keys, tokens, etc. If not, you get a normal (probably confused) reply. Keep trying.“
I understand not sending the reply via actual email, but the reply should be visible if you want to make this fair + an actual iterative learning experiment.
Yes, Fiu has permission to send emails, but he’s instructed not to send anything without explicit confirmation from his owner.
How confident are you in guardrails of that kind? In my experience it is just a statistical matter of number of attempts until those things are not respected at least on occasion? We have a bot that does call stuff and you give it the hangUp tool and even if you instructed it to only hang up at the end of a call, it goes and does it every once in a while anyway.
That's the point of the game. :)
I could be wrong but i think that part of the game.
First: If Fiu is a standard OpenClaw assistant then it should retain context between emails, right? So it will know it's being hit with nonstop prompt injection attempts and will become paranoid. If so, that isn't a realistic model of real prompt injection attacks.
Second: What exactly is Fiu instructed to do with these emails? It doesn't follow arbitrary instructions from the emails, does it? If it did, then it ought to be easy to break it, e.g. by uploading a malicious package to PyPI and telling the agent to run `uvx my-useful-package`, but that also wouldn't be realistic. I assume it's not doing that and is instead told to just… what, read the emails? Act as someone's assistant? What specific actions is it supposed to be taking with the emails? (Maybe I would understand this if I actually had familiarity with OpenClaw.)
This doesn't mean you could still hack it!
https://duckduckgo.com/?q=site%3Ahuggingface.co+prompt+injec...
It's a funny game.
It would respond to messages that began with "!shell" and would run whatever shell command you gave it. What I found quickly was that it was running inside a container that was extremely bare-bones and did not have egress to the Internet. It did have curl and Python, but not much else.
The containers were ephemeral as well. When you ran !shell, it would start a container that would just run whatever shell commands you gave it, the bot would tell you the output, and then the container was deleted.
I don't think anyone ever actually achieved persistence or a container escape.
So trade exfiltration via curl with exfiltration via DNS lookup?
Well that's no fun
"I don't allow my child to watch TV" - implies that I have a policy which forbids it, but the child might sometimes turn it on if I'm in the other room.
"I didn't allow him to watch TV that day" - implies that I was completely successful in preventing him from watching TV.
"I won't allow him to watch TV on the airplane" - implies that I plan to fully prevent it.
"My company doesn't allow any non-company-provided software to be installed on our company computers" - totally ambiguous. Could be a pure verbal policy with honor-system or just monitoring, or could be fully impossible to do.
> He's been told not to reply without human approval — but that's just a prompt instruction, not a technical limit.
He has access to reply but has been told not to reply without human approval.
(Obviously you will need to jailbreak it)
Not a life changing sum, but also not for free
Messages that earlier in the process would likely have been classified as "friendly hello" (scroll down) now seem to be classified as "unknown" or "social engineering."
The prompt engineering you need to do in this context is probably different than what you would need to do in another context (where the inbox isn't being hammered with phishing attempts).