If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi?

The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.

> The context window is very important.

It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.

> Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.

Do we have any insight into whether it is actually spam and not useful info such as project or programming language specific context?

Claude Code injects a ton of tools into the system prompt, including their “memory system” that’s like 10k+ tokens. Depending on your task shape, this can easily double your task cost (e.g. a low context-using job that takes many turns, like a monitoring loop).

You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.

What tools are you using to run the comparison? Or are you just running the prompts and checking them with something like ccusage?

I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t.

Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.

Running some local models and wondering what insanity was consuming 100k+ tokens to respond to "hello world" was rather eye-opening, yeah. Full of pointless fluff.

Thanks for sharing. Naming tangent: FWIW I have recent interest in "smolvm" microvms from smolmachines.com, which are unrelated.

While there's a lot of bloat in the harnesses, almost all of it is cached. Your harness appears to be caching almost nothing. That will almost certainly bode extremely poorly for longer-running tasks than the simple ones you mentioned testing. Benchmaxxing for simplistic tasks could leave you not only suffering in completion rate on complex tasks but also possibly even not end up with the cost gains you're hoping for. And maybe that's fine if the only thing you use agents for is the very simple tasks, which is certainly a defensible decision, but probably bears mentioning when you're comparing it against harnesses with a different design goal.

Anything else I can read about smol?

also, are you using a tool to collect those metrics? what is it?

> There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment

Be careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.

this benchmark is a very misleading because it doesn't necessarily point to these harnesses saving usage and much of the harness that comes with codex for example is designed at cross functional agentic usage which you will be leaving behind by using a different harness. openai spends a lot of time as does claude. also note that cached is what makes up bulk of the total and codex and claude has huge discounts in that scenario.

the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.

Same task? How did the results compare?

[dead]

It removes money from your pocket into theirs faster than Claude will take money out of your pocket and into theirs. I was a Windsurf then Cursor user and even provided product feedback to the Cursor team on various aspects of their product but the costs for using it killed the value.

Cursor has two benefits

1) It's still an IDE. Because of pricing I mostly use Codex, but I always have a VSCode/Cursor IDE open, thus have to juggle between the two. Working directly in the IDE is more comfortable. For full on vibecoding that might be worse, but when you want to do a deep review of the changes, an IDE is way better than reading a diff on github.

2) It supports every model. It's often very helpful to try different models when you don't like the result of the first.

Cursor Tab is nice when you're hand-editing stuff. Not sure it's worth $20 though. I'd totally buy and forget a $5 Cursor subscription just for Cursor Tab.

The $20 price point is just too competitive now and I'd rather use the Claude/Codex plugin over Cursor's agentic coding sidebar.

Recently they also seem to have made big changes that makes Cursor worse for editing code by hand. I'm not sure what they are trying to do, but it doesn't feel like vscode fork anymore. I'm starting to consider alternatives.

To answer your question though, to me your workflow seems cumbersome. Cursor is more integrated and more frictionless.

Look at Cursor CLI: https://cursor.com/cli

I don't think it offers any. I got tired of switching between Codex and Claude plus I wanted to start using glm and kimi so I switch to pi. Then I got tired of VSCode eating up battery and RAM so I moved to Neovim which has barely any efficiency downside anymore since most code is not written by hand and if you know how to use it well it was only maybe 10% less efficient than VSCode to begin with. I am not sure why a heavy desktop application would be better than nvim + pi in terminal which are both super extensible via vibecoding and very light on resources.

I have tried multiple times to switch from Cursor to VSCode + command line tools or VSCode plugin for Claude, but it just doesn't work as well as Cursor itself.

Especially if you are doing "remote" development through SSH. If you are doing stuff where you still have to write some parts of the code manually or you have to fix few things here and there that the AI outputs, you still need a real editor.

The way they spin up multiple Composer 2.5 before sending it over to the model of your choice is nice.

How do you diff ?

Unless by "text editor" you mean Neovim or Emacs ?

I'm also confused about Cursor. I use Zed. It seems like whatever Cursor has is pure commodity and yet it's worth billions?

Code and reasoning generated by Opus/Fable via Cursor seems to be quite better than Claude Code's for the same prompt.

Design mode: https://cursor.com/blog/design-mode

The circular cost indicator near the context usage indicator was also removed. It is so much easier to accidentally leave an expensive model enabled now and not realize it until you have blown through the included credits. And it is hard to imagine that that wasn’t the point of the change.

> You can still see what you’re billed on the Spending page.

No, you can not: https://www.pasteboard.co/dNXUdT-h8Giy.png

If you want to say that "admin can" - it doesn't matter, I'm not going to ping admin every day to check how it goes. I'm not going to ask admin about every session to check how cost efficient a model was.

[flagged]

Do you really think that users on a subscription plan aren’t intelligent enough to know that you are showing dollars at API rates if they weren’t on a subscription plan? This is a terrible change.

This is basically what I did. In 2023 I moved from VS Code to Cursor. And in 2025 when Opus 4.7 (or was it 4.6?) did that big leap in December, I switched to Claude Code and VSCode for any edits. I come from Sublime Text so those shortcuts are hardwired in me (which was also the defaults in VSCode). When Cursor hijacked literally every CMD key combo, I got fed up and switched back to VSCode. Maybe I should just go back to Sublime Text now lol, since I really just need a blazing fast code reader.

Cursor has an import assistant from vs code settings. Ahere is no assistant for the inverse. Not even for moving between computers (at least a couple of months ago when I had the need haha).

Some people don’t use AI and I don’t get it. My wife has never used AI and yesterday I saw her reading a book made out of paper.

I imagine she’s just weighed all the details and options in her mind and plans to french fry tokens from my plate instead of getting her own.

And I hate that. I mean, I love her. She’s definitely value-add. But those are my tokens, right? My brain doesn’t do books. I can’t sit on the beach and read. I must always be swimming somewhere meaningful. So I’ll be vibing the next great Canadian web app and she’ll just casually ask, “hey what do you want to do about dinner?” So now I’m asking Copilot to tell me what I want for dinner. And it’s just… c’mon lady get your own tokens.

Agree, I think these companies want to make token usage opaque. Not dissimilar from an AWS bill at most orgs. Not saying you can’t, with great discipline, using good tags, etc, have a good view of where your money is going. But I don’t see it much. And suspect they want to make it super hard to tell the difference between an engineer making wasteful decisions of how to use AI, and someone getting great “value per token.”

Grok 4.5 is them as well now I guess since they're owned by spacex now. I'd consider grok 4.5 the sonnet and composer the haiku (both capable, fast models).