Top
Best
New

Posted by matt_d 12 hours ago

HarnessTax: How Much Does the Harness Matter for Coding Agents?(harnesstax.github.io)
122 points | 51 commentspage 2
lexicalmathical 8 hours ago|
If this is mostly because of the size of the system prompt, then perhaps in long horizon tasks the "tax" will be less obvious.
sn0n 4 hours ago||
It matters about as much as where you leave the electrician, he’s gonna use what tools he has to get the job done with what he has.
mikert89 8 hours ago||
As the model gets smarter, you need to tell it less
big-chungus4 2 hours ago||
It goes the opposite way too. Small models need minimal harness like pi or they get overwhelmed by long system message and tool descriptions
NewJazz 8 hours ago|||
As the black box gets bigger, you need to stop expressing yourself even more.
malnourish 7 hours ago||
Perhaps, but then the value of each thing you tell it goes to.
jswelker 9 hours ago||
Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality.

"Why pay the waste disposal tax? Dumping into the ocean is free!"

Pi actively omits any sort of guardrails and sandboxing in the name of speed and simplicity, so it is not shocking that it is faster and simpler.

Doubling the cost of something in the name of vague security is standard operating procedure for big enterprises, maybe even quite cheap.

roywiggins 8 hours ago||
It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens.

(I do think Pi should ship with more than zero builtin sandboxing though)

zackify 7 hours ago|||
I have a couple hundred line TUI that just clones an LXD container and I get all my repos and infra isolated in 5s with pi having isolation to that container. Swap between any instance. Port forward to host all automatic.

Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated all in one tooling.

skirmish 3 hours ago|||
I tried putting my agent in an LXC / Incus container but then I also needed it to process videos stored on a NAS, and that was not easy at all. I tried a FUSE-based NFS client [1] but it was very flaky and would randomly report files having zero length and fail processing.

[1] https://github.com/facebookarchive/nfusr

KingOfMyRoom 6 hours ago|||
Do you have any links that would explain this?

I'm looking at changing my AI harness and am definitely interested in this

seanmcdirmid 8 hours ago||||
You can sandbox with tool hooks, at least in antigravity, I think a lot of other harnesses support similar features. Sandboxing via the prompt is just dangerous and unreliable.
imtringued 2 hours ago|||
The developers of pi.dev are geniuses.

They add a --tools flag, which can only add tools onto the built-in tools, so if you wanted to sandbox pi.dev by adding sandboxed versions of the existing tools you will have to run pi.dev via

    pi --no-tools --tools tool1, tool2, tool3
except..., that's not enough to sandbox pi.dev, because remember, it's a minimalist coding agent! So what does a minimalist coding agent do? Of course! It loads extensions by default and do you know what extensions can do? they can add default tools that bypass --no-tools!

So if you want to sandbox your agent, guess what you'll have to do? Yep, you have to supply --no-extensions.

Here is how to run pi.dev under its most minimal configuration under a sandbox:

    pi --no-tools --no-extensions --tools tool1, tool2, tool3 -e ./your-sandboxing-extension
Pretty neat, huh?

Sources: https://github.com/earendil-works/pi/issues/555

sandeepkd 3 hours ago|||
Wonder if you have any source to back that data, unless you are referring to the security code to detect the user and their usage itself to prevent against unauthorized usage patterns?
kouteiheika 2 hours ago|||
> Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes.

Um, no? Have you actually read Claude Code's system prompt and tools?

Here's Fable's system prompt: https://github.com/navanchauhan/agent-autopsy/blob/6d9c00e54...

There are a few lines of what I would call "for security and alignment purposes", but it's not the majority. And the majority of the bloat is from the tool descriptions.

The system prompt in this case has pretty much nothing to do with the built-in guard rails, that is: you cannot sidestep Claude's guard rails by editing/replacing the system prompt (which you officially can with Claude Code's `--system-prompt-file`).

shshsjsj 8 hours ago||
[dead]
robinpie 9 hours ago||
claude code feels mildly shitty to use in the way that every other vibe-coded-project-got-out-of-hand project does, which is like, not that bad, but it's fucking ridiculous for a 2 trillion dollar company's main companion product
nativeit 4 hours ago||
I’m still baffled as to where these 100x valuations came from. Amazon’s market cap is $2.65T, and I don’t see how any AI company comes within a faint whiff of that sort of value.
topaz0 6 hours ago||
It's pretty great to not use, conversely.
ethanprk 2 hours ago||
[flagged]
shivrajnag 6 hours ago||
[dead]
nirmeetimthebes 6 hours ago||
[dead]
pyfd 4 hours ago||
The part of my harness that earns its tokens isn't instructions, it's the bits that take claims out of the model's hands.

Two examples from daily use on a small business's systems. Telling it to batch browser calls in the system prompt didn't hold; a hook that denies the seventh consecutive single call did. Telling it to record which machine did a piece of work didn't hold either; having the CLI stamp the hostname did.

Both are nearly free in tokens, and that layer is where the reliability actually came from. The prompt-size argument skips straight past it.

appbeforelunch 8 hours ago|
[dead]