Posted by kevin_hu 1 day ago
Closed models might be better generalists because they don't have that much constraint on size and come from well funded labs that can afford to better train for generalization. If a model's actual goal is to be open then smaller is better.
By the time, we want to code, the scope has been limited pretty greatly to the point that my rule to the Ai is “don’t make any decisions outside of the plan, if something in the plan turns out to be untrue or cannot be implemented after 2 differing attempts stop and ask for clarification”.
Sub frontier level Ai can circle endlessly thanks to a bad assumption, but a human can cut through that by coming at the problem sideways.
The Opus frontier model will spin up a cheaper Sonnet subagent to explore the code base. Similarly we can have a Pareto optimal subagent write most code and verify it.
Sure some world knowledge like things are affected by gravity, how people function, how a society function I think is needed for a coding AI but not trivia.
Words that are or were fairly common in tech circles — "bailiwick", "gatekeeper", "daemon", "boundary", "infer", "method".
It's true to say that most of these words have some specific meanings in tech, but you don't necessarily want the LLM to have only those narrow meanings when you are talking to it. A broader grasp of the word means the LLM won't go down a specific pathway before understanding you by analogy. That broader grasp comes from non-programming understanding.
Words only have their meanings through general context.
First, OpenCode and Command Code offer you basically 6x-10x usage for some models (not all, though, look at the exact rates per model). That's a very similar model.
Second, while Anthropic subscriptions on paper offer the most generous usage per dollar spent [1], this comes with a few caveats. First, they are the most expensive of all models in cost/task. Second, if you use Claude Code non-interactively (claude -p "..." or SDK/ACP mode), they apparently charge you 70% more usage; you have to use their apps directly and interactively to get the full benefit of a subscription. Also, Anthropic has the lowest cache TTL at 5 minutes, which can really hurt for interactive use. Contrast that with OpenAI's default of 30 minutes (and according to a recent post by Tibo, apparently 12 hours for subscriptions, though I haven't verified it myself) or the fact that DeepSeek caches prompts for three days (which is a major reason for their extremely high cache hit rate and how cheap they are in practice), and it doesn't look like THAT great of a deal in comparison anymore. It is of course still good value for money, just not as great as it may look at first glance.
[1] At least before the recent usage bump to OpenAI's Pro plans, though that doesn't seem to include the Plus plan if that is what you are on.
edit: and of course from first party labs as well, like xiaomi, z.ai, kimi etc
Yep, on subscription plans DS4.1-Flash-1017 feels more or less infinite. I still reach for Astra at the high end of tasks but DS is great for cheap "repeatedly crawl the entire codebase and find query optimization" type tasks. I also use a lot of K3.1 Pro lately although admittedly it's a lot more expensive.
Also personally I find that the closed tiny models (Haiku, Luna) feel a lot less capable in practice than their benchmark scores indicate relative to DS.
That being said the closed labs own the high ground of intelligence, nothing really compares to Astra in my opinion in terms of sheer "it will achieve any task you give it," Kool-Aid-Man running through walls style capability. It's quite expensive but it's also very very capable.
If someone is in a position to use the personal subscriptions, the $20/month plans are about a dollar per work day. I'd rather have the $20/month plan for Opus or Sol 6.1 than deal with all of the extra churn and babysitting from Luna or Deepseek.
I've had really great results with cheap flash agents using a system that has a parent agent that spawns 2 sub agents. They remain active throughout the whole session and don't lose context.
The 2 sub agents get the exact same prompt and tools as each other and they independently answer but before responding back up to the parent they exchange their findings which must be backed up by a citation and then debate their findings and citations repeatedly until they can agree on correctness of all points. They are also heavily encouraged to never assume and always verify with the prompt and tools and guards where they can be realistically enforced.
It churns through tokens but tokens are crazy cheap with some of the plans out there. Could Opus one shot most of those same queries in like 1/4 time? Yep, but that's not the point for me.
DeepSeek Flash for parent then DeepSeek Flash and GLM 5.3 flash as the debating subagents gives best overall results in my experience.
For context I do equal parts sysadmin/devops and having them write software
As for the session maintenance, I have it configured to trigger a compaction when it ends a 'debate' with >25% of context used but before the compaction it puts what it thinks are key points into an MD file. The approach has worked so far in terms of stopping the agent from getting silly with too much context and retaining information. These flash agents get too flakey for my liking past 25%
DSH has been fantastic, the customisation is great, like I mentioned before. What has got me to stick with it though is the way it presents information like diffs, subagents, tool calls, thinking and sessions. It's nothing revolutionary but everything is where I want it to be and provides just enough information
As stocks fall and the investment money dries up, it'll be easier and cheaper to buy hardware to run local models
Let’s say I want to run qwen 27B or flash next on a machine I own, off solar panels that are also sunk cost.
How did they price the marginal cost of a token? Depreciation on the box + reverse billing opportunity cost with the power company?
This is totally suitable (speed wise) for coding.
This is not some top of the line hardware (might be now, but it should be run of the mill for people that built their system <1.5 years ago).
Seems very slow, but it's actually fine.
It is also evidence of commodicisation / brutal competition. I wonder if they cut costs because there is stiff competition from open weight models (and arguably we are in the blitzscaling phase where they try to corner the market, not the enshitification stage).
Where it differs from the larger models is that it has less knowledge; respectably as it performs for its size, 552B parameters are still a lot less than the multiple trillions that frontier models have. Sol/Astra/Opus/Fable may know the solution to the problem directly or to a very adjacent problem, while DeepSeek has to work its way through it from first principles and then sometimes fails (because LLM reasoning is really a heuristic-guided search). But this is also a failure mode that I have seen with Luna.
[1] As evidenced e.g. by ErdosBench. https://www.ulam.ai/erdosbench
[2] Using this as an example over software engineering tasks here, because it tests reasoning capability more than knowledge retrieval.
I'm using Claude all day at work and deepseek privately for very similar tasks. The price/performance difference is extreme.
What's more important is that open models can be inspected, they can be customized, and you don't have to worry that a company will decide that you're not allowed to use them for stuff like reverse engineering things or do security work.
That's why open models, and especially ones you can run locally will always be important. When you're renting closed models, you're basically a serf who has no say in how the model can be used.