> On our benchmarks, Claude Opus 5.5 leads in agentic coding, computer use, and knowledge work. That said, at these levels of capability we’ve found that benchmark margins have become a less reliable guide to real-world differences. In our own use, the gap between Opus 5.5 and Claude Fable 5.1 is narrower than these scores suggest.
In general, "benchmark margins have become a less reliable guide to real-world differences" sounds like a big problem. It was certainly the biggest problem with the previous generation of Claude models for a different reason, because the non-code output was nonsensical, and that is not being benchmarked at the moment. But I'm not sure what to make of this admission.
"benchmark margins have become a less
reliable guide to real-world differences"
sounds like a big problem.
1. Real-world use cases typically involve big, hairy, crufty, tech debt laden codebases and benchmarks do not.
2. AFAIK "success" in a benchmark essentially boils down to "do the tests pass and do we get the right result?" which is something the LLMs have been achieving with ease for a while, except maybe for uber-challenging coding tasks that would be outliers in just about any workplace. Whereas real-world software engineering is usually just a bunch of CRUD... and "success" involves harder to measure dimensions like "maintainability" and "did you overengineer this?" and "how did you cope with a bunch of vague and maybe contradictory business requirements?"
Having said all of that, I have never ever looked inside any of these benchmarks. I'm putting my guesses out here strictly in the tradition of "the quickest way to learn about something is to be wrong about it on the internet."
In my experience Opus 5 is the worst of all possible worlds, it's dumb and headstrong. It just runs away with tasks you didn't ask it to do, is reckless, and basically is unusable in my experience.
Not sure why but my guess is that this will be worse. Happy to be proven wrong.
I've really gone in the opposite direction: having a dumber model orchestrate. In my case, it's usually a Luna orchestrator spawning Sol/Astra subagents to do the "big brain" work of planning and reviewing.
Reason I went with "dumb orchestrator" was just to save tokens. Having Opus/Sol (let alone Fable/Astra) orchestrate was burning tokens like crazy for me even when much of the gruntwork was being done by Luna/Sonnet/Haiku subagents. (Luna is also really good, like way better than Sonnet...) Perhaps it was a skill issue on my end though, maybe I wasn't just managing context properly.
"Better" in every sense of the benchmarks and absolutely horrible results in my day-to-day work.
The verbosity, goal post moving, tendency to leave work unfinished, over focusing on unrealistic root causes when debugging, etc... etc...
It was the first time I actually pinned my models back because I just could not work with 5 for the price and performance it gave me. Hoping 5.5 is better this time around....
Yay, yet another model I can't use for anything interesting, even with CVP.
It still covers as much detail as Opus 5, and provides a healthy amount of caveats, but it's much better at just providing the answer up front and framing the caveats.
5.5 communicates like a real collaborator and not something that constantly demands my mental energy to decode what it's trying to say.
In the application I'm working on, we don't want "user's local midnight" to mean "next day". If a live sports game extends past midnight, we don't want the UI to group it under "Yesterday's games." So I asked Opus 5.5 "can we use the backend's API response as the definition of today instead of the browser's local time?"
Opus 5.5:
> Your assumption holds, and the commit history backs it up. Every time the frontend decided for itself what "today" meant, it broke and was later replaced with the backend's currentSeasonDay. There's one catch: trusting the backend only works if the client's copy of currentSeasonDay is current, and right now it often isn't.
Opus 5 would have buried the answer in some kind of unhinged lecture. Something like:
> The assumption is right about ownership and wrong about possession.
> currentSeasonDay is the authority; the commit history has already paid for that conclusion. Every local reconstruction of “today” became a second clock and was deleted. But naming one clock does not make every copy of its reading current.
> The remaining failure is on the other side of the seam: the frontend no longer invents the day, but it can preserve an old one indefinitely. The source is right. The observation is stale. Those are not contradictory states.
> Do not reopen the ownership decision to solve a freshness defect.
Basically the point is just poll the endpoint every 5 minutes or so