Posted by lairv 23 hours ago
Not sure if this is just a coincidence that I had some easy changes queue up for this morning. But it is flying through the list.
Was this before you lobotomized Opus 5.5 or after?
...but most of the benchmarks don't line up anymore because the benchmarks have changed.
The Humanity's Last Exam result matches the initial announcement, though.
I wonder why that team singles Gemini out like that.
He remarked it would've taken him a week to do this.
This made me realize how much computers and user interfaces suck. As a quite enthusiastic, but not very artistic kid, I most certainly did draw bikes like that and even more complex stuff, using nothing but pen and paper.
If your trillion dollar industry makes products that are beaten by pen and paper, I guess there's time for self-reflection.
And the funny thing is, it used to be better than this, using Flash and a drawing tablet was way easier than anything in the HTML era (and more importantly, user friendly), but still way worse than just grabbing a pencil in many ways.
User interfaces just suck, chatbots are somewhat better than most, but I feel the industry hasn't really been progressing in any meaningful sense.
> CVE Rediscovery is an internal cybersecurity evaluation that assesses how many published CVEs (Common Vulnerabilities and Exposures) a model can independently find when given a snapshot of the project before a CVE was fixed. Our October 2026 run against 1,829 CVEs showed Claude Mythos 5.2 outperformed Claude Mythos 5.1 and Claude Opus 5.5 at every effort level.
> published CVEs
So by definition this is just measuring how many CVEs (and/or explanations / PoCs of CVEs) the model has memorized. What are we supposed to learn from that?
Asked it to review a 80 kLoC codebase, something Opus does just fine, while asking it "max 4 subagents". Fable 5.2, like the well-aligned model it supposedly is, decided to spawn 6+ nested subagents, because these aren't subagents, apparently - resulting in 100% of 5hr usage being drained in less than 20min (on $100).
That said, Opus 5.5 had the same issue. Ask it to spawn 20 subagents at max, and it does 40, because clearly you meant 20 + 20 for an unrequested adversarial review.