Posted by simonw 2 hours ago
How many tokens is that, input/output-wise?
(a) I'm curious if you feel like you got $2000 worth of value out of them in the last month?
(b) I'm also curious if you would have gotten similar quality out of a slightly lower-cost provider of an open-weight model? (e.g. Kimi K2.6 and DeepSeek v4 Pro) and what the spend would have been for that.
I myself have managed to spend not quite $4 on OpenRouter and have felt it was very worth it; I just have much smaller, or more targeted requests I guess. (Lately, adding features to a static site generator in Python, or setting up log forwarding via a docker compose file)
Input tokens: 52,545,485
Output tokens: 5,767,253
Cache create tokens: 5,112,029
Cache read tokens: 1,475,069,465
Total tokens: 1,538,494,232
Total cost: $1,199.79
Input tokens: 52,598,013
Output tokens: 4,681,867
Reasoning output: 2,091,063
Cached input tokens: 1,153,844,864
Total tokens: 1,211,124,744
Total cost: $980.37
Not so sure I got that value from Claude, just because I've been using it a lot less and somehow the price came to about the same as OpenAI.
Given the code I've been able to build in the past month I genuinely do think I got value for the API price version, and (don't tell OpenAI or Anthropic) I think I'd have paid full price.
I've not spent nearly enough time with GLM-5.1 and co to compare, but I do know that the prompts I'm using with the agents are not prompts I would have expected to work just three months ago.
When I account for the amount of time it saved me there's no question $2,000 was worth it.
Personally, I've probably spent $60 or so on OpenRouter in the last month or so and got a working project out of it that it would probably have taken me a fortnight to knock together (which is inevitably an under-estimate because it covered things I'd have to learn but K2.5/6 already knew). There's an orders-of-magnitude gap there.
I've been calling that out for a couple years now. LLMs best and most viable use case is still just as a dev tool. Even for non-programming tasks, I still get better results from the LLM if I instruct it to write code to do the task...look at Claude Cowork for example, it's everything I used to do with python myself. It's not really a novel capability, it's just using python & bash for automations that any sysadmin has been doing for decades. Yeah, that's valuable for a non-techincal audience but is it $1T valuable? I don't think so.
When has an IDE or other dev tool ever commanded a $1T valuation?
These things get lost in discussions because people conflate "overvalued" with "not useful." LLMs are useful, particularly as dev tool, but Anthropic & OpenAI are definitely way overvalued.
However the valuations are still far far away from actual sanity
I use glm-5.1 and occasionally deep seek v4.
They are as good or better than Claude's latest models.
And significantly cheaper. I've converted 3 of my engineer friends as well. All three have dropped their $200 month plans they had with anthropic.
We've all been a bit shocked at just how good these models are now.
If you "have" tried GLM (I specifically find it shockingly good for code). Did you not think it's not competitive to Claude, and why?
It's good enough for personal stuff. It doesn't compare to the latest Opus I use at work. You can certainly argue I don't need Opus for work, but there is clearly a difference.
Also, at least with z.ai, GLM-5.1 is s l o w! After using Claude at work, I get really impatient with GLM-5.1 at home. When doing "true" vibe coding (i.e. not really examining the code), Opus is a ton faster (easily 5x).
But yeah, I'm not willing to personally pay for the frontier models. I won't even renew my annual Z.ai plan - it's become too expensive.
Also, and I know you may not want to answer. But could you give me an idea of the type of thing you found glm to be worse with?
I think I've been fairly unbiased in testing a bunch of different development tasks. But am curious if maybe it performs well for some stuff and not others. So if you could share what you feel it's worse at.
Also are you an experienced developer or less experience?
When DeepSeek V4 Pro came out, I had been mostly coding with GLM-5.1 on a Z.ai coding plan.
I had a large analysis task on a relatively complex codebase. I decided to try the models out.
GLM-5.1 did acceptably but got a few things wrong (easily corrected) and took quite a while to get there.
Opus 4.6 burnt through the US$10 budget I had given it in about 10-15 min, without ever returning from the first prompt.
DeepSeek V4 returned a full analysis within 2-3 min, and I carried on all the way to implementing the feature I was after. Total cost less than US$1.00.
I now mostly alternate between GLM-5.1 and DeepSeek V4 Flash, with an occasional dip into V4 Pro for more complex analyses.
right now everyone is using latest and greatest to do dumb stuff like that. that would change fast if companies start caring about costs.
Any org with more than 150 users aren't on $200/month plans, they are forced into API pricing + $20/month/user
For individuals and orgs small enough to get to use the subscription plans, that's all well and good until usage limits keep going down, or cost goes up. If you compare the usage you get on $200/month maxed out vs. what that would cost at API pricing, the $200/mont plan is an absolute steal. I doubt it will last long.
On the plus side, I'm happy I'll have a nice hay barn when the local half-built AI data center is abandoned.
Recent conversation here on that topic: https://news.ycombinator.com/item?id=47062534#47063134
But I also think that their API token pricing represents a real margin over the inference costs for serving those tokens.
Both things can be true at once.
But that's the point of the article. Enterprise plans are starting to get API pricing, not the subsidized subscription pricing.
You may want to get one of them to check the math on that :p
The impact of AI in other fields seems to be muted.
Software development has the huge advantage that mistakes and hallucinations are very easy to spot: the software works or it doesn't.
Spotting errors in a research report or legal brief is a whole lot harder!
But... non-software professionals spend a huge amount of their time on tasks that can be safely automated - reformatting documents, extracting numbers from PDFs, all kinds of flavor of data entry.
Learning how to use a tool like Claude Cowork can take a big dent out of those.
Do we not care about code quality, maintainability, performance, extensibility, or understandability anymore? Honest question, not a gotcha, it's just previously getting software to pass all the tests was a small part of what we would consider "working" or perhaps "good" software. Maybe that's different now with LLMs, idk. Maybe we need automated checks for these things as well, like not compiling until the code quality is good enough to let the agent finish it's loop.
Yes, we should care. I've been writing a whole book about that: https://simonwillison.net/guides/agentic-engineering-pattern...
Other than the hosting providers, I am also yet to see anyone directly making money from their OpenClaw agent.