Wow this is great, it explains what I've seen. My Codex sub seems to last way longer than Claude in real codebases, and Claude eats up a ton of context in the initial read. I thought the harness might be the cause but it seems like tokenization is probably the bulk of it.

The "Honest size of the effect"

Honestly, haha

The real price is how often a model uses subagents that scan your whole repository multiplied by thinking level.

It's a terrible metric, it's kind of like companies paying employees by the hour for white collar work.

Most people here probably don't know what it was like to work a contract job and being paid based on actual deliverables.

The incentive of AI companies is to create as many tokens as possible to solve any given problem. Just like your incentive as a software engineer is to create as much complexity as possible in order to use up as many hours as possible.

This is why big tech companies have millions of lines of code... They've got thousands of engineers rapidly churning out tokens.

The difference in number of tokens I use in my day job vs side projects is massive. You can see the inefficiency quantified.

Show me the incentive, I show you the result.

This piece focuses on the cost differences from the tokenizer, which do matter, but I wish they emphasized more that even adding the tokenizer to your calculation doesn't provide you with a good way to calculate cost for agentic coding tasks.

Other traits where models differ that have an even greater impact on your total spend:

* How much context do they load in to solve a given task?

* How long do they spend thinking to get equivalent results?

* How many times do they stop and ask you for input, and are you there to respond to them before the cache runs out?

* Etc.

Incorporating the tokenizer just makes a very imprecise measurement of cost a little bit more precise, but in my own experience I have not found that the token cost is a significant driver of task cost whether or not you incorporate the tokenizer. Everything else about the model's behavior has a much larger impact.

Yes, and verbosity (thinking) is a huge factor.

Are there any advantages of the new tokenizer? Does it have a larger or smaller vocabulary or just differently weighted?

This article doesn't address the inference side cost. Not all tokens cost the same. If the response is predictable you get a few output tokens for for the price of 1. The further an output token is the cost of generating it grows linearly due to attention.

this is surprisingly high delta. to make matters worse, reasoning tokens account for the majority of tokens and they are completely opaque so it's hard to tell how much of that is prose or code

Just start pricing in bytes input/output. This whole "token" and "tokenizer" thing is an implementation detail that shouldn't even be leaking out into the API.

Providers change tokenizers all the time with model updates, and it's often not even possible to query/figure out how text is tokenized without actually just sending the LLM a request.

Just switch to charging for bytes of intelligence. Please. Claude Shannon figured this out decades ago.