It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.
I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level
This happened to me ages ago with Opus. I added a note to the agents file saying that explicit user instructions in chat override all prior instructions and I've not had the problem since (now using Sol).
Weird, I also use Sol (medium) for a personal project, and I had no problems with those things. I simply tell it that something changed, and it happily edits everything to make that fit. When I tell it that something was verified by a human, it accepts that as well.
I also told it early on (the first spec was mobile first) that my main usage is on the desktop and mobile is secondary, it happily accepted that once again, and the most accessibility thing it had done was making sure contrast didn’t totally suck on a greyed out row.
Considering your last sentence, maybe high and x-high have those problems? I didn’t test them.
Random off the cuff comments or one off instructions can get recorded.
And from then on they are often treated as carved in stone commandments.
It will glom onto the tiniest thing and extrapolate from it.
A .github/CODEOWNERS file seems to help when it’s going down that path, but I don’t like to indulge it..
So your idea might work.
Could this be fixed with better harness restrictions/tool sandboxing?
In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.
For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.
If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.
As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].
[0]https://github.com/harbor-framework/terminal-bench/blob/v3.0...
Also, a skill like grill-me from Matt P. https://github.com/mattpocock/skills.
One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.
This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).
I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.
For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).
Once the specs are being completed and splitted into beads, I span multiple agents (ultreworkers) and as part of a contributing guidelines I specify to use gitflow + git worktrees, then pr.
"Cheating is often more efficient"
First one of these I've seen using DOM manipulation and CSS transitions instead of canvas, so that's neat.
I asked it to modify our cicd workflows so that only a select few can raise PRs against them. Opus took 15min and added a banner to every file and did a few other things. Then I asked it, see you added all that and still since the last 2 commits you have modified the file. So whatever you did is useless
It "thought" for a second and then said that I was right