Design: https://image.non.io/78795662-8bfc-4e14-8d72-3738392aa6b3.we...
MiMo 2.6 Pro Ultraspeed (36min): https://html.non.io/annui-mimo/
Grok 4.7 (25min): https://html.non.io/Annui-grok/
Astra (19min): https://html.non.io/annui/
Overall this felt like the weakest of the three. Ultraspeed was fast as far as tokens per second goes, but it overthought quite a lot of things resulting it in having one of the longest build times. That overthinking didn't lead to better results either - note the statue with the cropped off arm. It's also the worst implementation of the dynamic lighting effect / displacement effect - the background especially has some significant distortion. Astra was the only one that seemed to understand that displacement should happen less the further something is in the distance.
Here's a vid of all 3 side by side with the source design: https://non.io/video/annui-comparison.mp4
Perhaps instead of giving it a one shot task with vague prompt. I wonder how it can perform with much more detailed and constrained prompt (can you please elaborate more on the level of detailness and ambiguity that the prompt is and where does this model seem to overthink the most?)
Also are there any ways to tame such overthinking of models in general?
I hope that once models start becoming smart enough (I think for me it’s already there) or becoming genuinely the Sota. They then start focusing a lot more on optimizing token usage
Staggeringly low for a frontier model.
KillSwitch-Bench 1.0
Claude Opus 5 66.9
GPT-6 Astra 57.9
Claude Fable 5.1 46.7
MiMo-V2.6-Pro 38.8
Muse Spark 1.3 36.5
- capped per-task budget and time limit
- No internet access
- different harnesses mixed
I would argue that this benchmark is uniquely suited to how most people use LLMs because it actually tests common harnesses and it is a true coding benchmark for a language that is unseen, thus testing the LLMs actual ability to understand nuance and learn.
Internet access is restricted to ensure that over time models cannot just look up the source code of KillSwitch, which would allow them to cheat.
In my experience models just don't take forever to mark tasks as done.
For my coding usage I don't set time limit so benchmarks that do so are providing me less interesting use cases.
With that said, I can understand having budget limits for expensive LLMs for those of us that don't have infinite VC money.
As for no internet access, the issue is that for most problems we do want Llms to be able to search docs, API specs, GitHub issues, etc. So not allowing that usually just favours larger LLMs that were able to memorize more data, not necessarily smarter ones when both have internet access.
Im a bit lazy and only use the free models different companies host and the biggest difference i see is that some models (Gemini, OpenAI) get progressively stupid in long chats. You end up having to start a new session every oncr in a while. Or they get really hung up on a theme and cant shift to a new topic.
By contrast, Ive been impressed with Qwen. I have some chats on research and code architecture that have stretched for weeks without any noteable change in quality (though occassionally it seems to "rush" to an answer)
Im just looking at all the listed benchmarks and im unsue which i should be looking at
And later they further cut Sol and Terra pricing by 20% (maybe only in the API) and Luna by 80%.
In fact Luna still outperformed DeepSeek Flash 4.1 in cost per task on Artificial Analysis when I last checked.
However, Luna is slightly less intelligent. I have a feeling that it's pretty dumb and prone to hallucination unless running at xhigh or max effort, where it somehow manages to work quite well.
I did not personally test the open weight models beyond the old Qwen 3.6 27B, which produced unusably bad results for me.
The competition is great, and I hope Chinese models will continue to force leading US labs to offer models at a low price point.
That said, I don't think the Chinese labs have anything over OpenAI and Anthropic when it comes to capability or efficiency - I have no reason not to believe the US labs have even lower cost to serve the models.
So you don't have much perspective on things, it seems. Let me introduce you to the GLM 5.2 and then 5.3/5.3 flash series of... "oh, wow, I should have bought some RTX PRO 6000's while they were 'cheap'" stage of progression.
As someone carrying multiple max subscriptions to both claude and codex - primary workhorse is glm 5.3 flash running on rented GPUs for less than a latte/hr.
I also found qwen 3.6 27B nearly useless for my own needs. DS4 flash 0731 and then 4.1 have been nearly as eye opening as glm 5.3 flash, but have their own warts.
Do also remember China is this far in the AI race despite all chip restrictions from America. If they were in equal standards I truly think Chinese models would have long surpassed American ones. Also would like to remind how Anthropic CEO is being hostile and blaming Chinese models with distilling meanwhile their own models claimed to be Qwen¹ and their stance against open models is negative² and they still keep blaming China for it.
1- https://news.ycombinator.com/item?id=48671252
2-https://www.anthropic.com/news/position-open-weights-models
Why wouldn't he? If there really was 25,000 accounts breaking ToS any CEO would at minimum be upset. Evidence of Claude distilling qwen would be damning but that a) makes no sense b) doesn't exist afaik.
Given the difference in compute, it seems plausible.
However, the researchers at the US labs are surely no less talented, and they have better access to hire talent globally.
They too have to serve their models efficiently at a large scale, and with current capacity constraints this must be a top priority.
OpenAI reduced prices and Anthropic increased weekly usage limits.
One simple task: I needed an LLM to go through and clean up a few thousand page descriptions and titles in my personal search engine index, where the human web page authors had put in no effort sigh. I did a shoot out between Claude, Luna, GLM 5.3 Flash and Deepseek. Despite the high cost, Claude's descriptions were terrible, and even Opus warned me that the descriptions coming back from Haiku were "generalized, not accurate". I expected I would choose Luna because of price, and occasionally it did have wonderful descriptions (one captured emotion in a way no other model did). But in the end, the GLM 5.3 Flash descriptions were the easiest to read, they flow well while also being accurate & including necessary keywords, and being highly affordable. So it won out. It's a task that is nowhere near frontier, but a task where somehow China is better than frontier.
debatable if a turn around is possible before '29
Not an enemy, just a danger.
"plurality" would have been accurate over "half" on my part
But at least I can run Chinese models locally, and strip a lot of that censorship/refusal.
So yes, we are getting scammed by American SOTA.
Just tried 2.6 flash on a really niche topic I specialise in and it has done a really good job. They’ve definitely polluted their training data with claudeslop, but looking past the slop there is a decent model.