Posted by kevin_hu 1 day ago
I don't think I am going to try to beat them at this game anymore. I've got a machine in my closet that is both collecting dust and accruing value. A very maddening state of affairs.
At this point, trying to beat a major provider is like running your whole house off grid when the alternative is 12c/kWh that is already wired to the premises. You might win if you make it your only job for years on end and have a very specific lifestyle that is tolerant to weird tail risks (assuming you aren't willing to go back to the grid in an emergency).
So what's the point? After the codex <> web quota merge next month it'll be useless ...
I'm running full swarming on a 90$ subscription lol
Using DeepSeek I've made the Wayland compositor I use every day. It's finished now in the sense that it doesn't crash or have any bugs, nor can I think of any features to add or anything to change.
When somebody else is paying, I'll absolutely just throw Opus 5.5 at it. Who cares. It gives the best chances that the output is what I want, at no extra cost to me. But for my personal projects, using DeepSeek in Pi with a workflow extension is competent.
Sorry for throwing lots of text at you. All of this to say there's more to it than just which model is more intelligent. There's also speed, cost, and "good enough". Perhaps the problems I'm working on right now are just not complicated enough for "Opus is better" to be a real selling point.
Open weight models are generally more humble when working at fronteir, seek human input more reliably on thi gs they have no buisness deciding on, and can be trusted to not fuck up codebase in some way. Also, the way claude talks by default and tends towards even with all the "use this standard or that" tricks, grinds my gears in a way that drive me postal.
So, all in all, the shit prose, the constant oversight robbing me of attention better spent elsewhere, and high cost, and seemingly correct and difficult to catch errors on first pass make avoidance/non-reliance a nobrainer.
When dumb models are right, they're right, because likely its a solved problem elsewhere, and when they're wrong its quite transparent. Claude and fable are extremely good at spinning resilliant bullshit that can take quite some mental labor to verify and disprove, usual problem is flawed or deeply i correct assumptions on top of which then relativly sound reasoning and evidence is layered, or otgerwise weird contradictory decisons made "because i thought it would be better this way" that waste 48 hours of gpu time on an invalid model because claude fucked the one small part entrusted to it, which was extremely well defined and reasoned/backed by literature and it just made an executive decision to deviate without consulting or warning about it.
so fuck claude.
At least Gemini 4 Argon has made writing the front ends I hate and don't care about a breeze.
I have no idea how much that's worth to other people, but it's more than worth the account or three I use.
But I think it becomes a little more relevant when making a price-performance comparison such as the article makes between “open” models and “closed” models.
Because the reader is invited by the term “price-performance” to think about efficiency, when in reality the results are being skewed by the fact that the user is not the only one paying for the service: investors are chipping in fairly directly for every token, at inference time.
Users are going to continue to do their own thing for their own reasons, and they should; that’s the market at work. But this is not an efficient market where meaningful comparisons can be made; all of the major closed providers are loss-leading.
For us who spends more than $150K tokens a month with less than $5K spending in subs, it's really hard to justify the switch and it's even senseless (for now).
I had a check on artificialanalysis.ai and it shows a 16% reduction in token and 200% increase in time for Opus 5.5 Max compared to DeepSeek V4.1 Flash Max. I don't think the data is very accurate for my real life workload, but for any workload it represents I would think Opus having little advantage.
Open-weights excel primarily at coding.
Let's scale this out 3 years into the future: - Opus 6.7 costs an arm and a leg to run for 2 prompts - Astra 7.89 literally requires you to buy-now-pay-later because it's so expensive - As a bootstrapped dev, your only option is serving local Qwen-4.04-27B that runs on your old RTX (that you bought in 2024)
Sure, it's worse performing. But if you can't afford the best ones in the first place, what's so bad about having a "pretty decent" model that you run essentially free (only paying electricity) if it does what you need it to?
... and if we're all in the permanent underclass anyway, what does it matter that your local model is dumber than the SOTAs? At least you have one! Some people who can't afford GPUs still have to type emails by hand.
(jk, this is actually a cool dataset though ^)