Posted by lairv 22 hours ago
I would imagine Anthropic is hitting a wall on the number of simultaneous agents that can be useful on a task and are working on increasing that maximum. I would assume if they could get twice as much progress by using 10000 simultaneous running fable instances instead, their brief would state that.
IMO, it is very interesting and also very alarming. At some point the barrier to the exponential progress is the amount of computation available. The average human is limited in the number of 'human instances' he or she can interface with at one time.
if 1000 fable instances is better than any human at a task, how about 1 million instances running? Much easier than finding and hiring and coordinating 1000 extremely productive and intelligent human beings. Much cheaper too, it appears.
We are very close to some very scary things happening. I just recently came to terms with the fact that an AI could code and manage coders faster and better than me, I don't think anyones is ready for what is about to happen.
More broadly, what distinguishes humans from AI is what makes humans dangerous. Our society tolerates the danger humans pose, as it's considered a prerequisite for a free society. But our society would never tolerate this danger from a tool.
So if I run a script that's unethical? How is a LLM inference loop different than a script?
> But our society would never tolerate this danger from a tool.
Sure about that? Cars kill people, chainsaws kill people, heck, a lot of useless tools such as alcohol kill people. We tolerate dangerous tools just fine, there has never been a tool where humanity would be like 'nope, its cool, but its dangerous so we are going to stop using it'. Not biological weapons, not cars, we never go back.
ML and DL have their place in research which can push the frontiers of the problems and solutions know to humans today. But are the majority if the companies focusing on that? Nope.
Any technology is a tool and if it is in the wrong hands and there are no guardrails in place, both the human and the technology become dangerous. A car in th hands of a robber, a chainsaw in the hands of someone who has an intent to hurt someone,etc.
As for tools in the wrong hands and all, sure. Obviously. My point was that humanity never abandons a technology because of risks, or real issues.
gpt-6-luna XHIGH as the reviewer agent still found plenty of bugs that Fable 5.2 missed!
Here was Fable's summary of the session:
The reviewer ran 16 reviews over 9 commits this session (7 rejections, 9 required fixes, all valid); three of them, on the plan, changed how the numbers are computed.GLM-5.3 almost equal even I've tried it and it is proven for security audit in the blockchain realm
If we compare medium or high, then yes, Fable 5.2 is significantly better than Opus 5.5. But it's better not because it's much smarter, but because you can see it's allowed to spend more tokens. At high, Fable 5.2 wrote 76,158 tokens for $3.05. Opus 5.5 wrote 19,767 for $0.40. If you compare by token count, it's just spending more.
The most interesting one was to compare them at max. Streamed, Fable 5.2 spent the whole 128K output ceiling on thinking and returned nothing. Given 300K on the Batch API, it finished at 250,082 tokens. Opus 5.5 Max needed 255,957.
It turns out it's basically the same as Opus 5.5 Max. I don't see much difference. There's a little bit of a difference, but I think Opus 5.5 actually drew slightly better and is twice as cheap: $5.12 against $10.00 at the standard price. If you look closely, the icons are extremely similar. Fable 5.2: https://playcode.io/marketing/assets/static/claude-fable-5-2... Opus 5.5: https://playcode.io/marketing/assets/static/claude-opus-5-5-...
It really looks like they're just tuning the effort level, that's basically it. At different effort levels, they just gave it more context, and the model got smarter.
For now, my recommendation is still to do everything on Opus 5.5 Max. The model won't write that many tokens in agent mode. It only does that when there are no tools at all.
For Anthropic's IPO, I expected a larger version of Opus 5.5 with great capabilities.
This isn't that. It's barely better at all, to the point where it's likely not even worth using with the speed and cost trade off.
I thought the Opus 5.5 improvements would scale to a larger model, clearly this hasn't happened. Scaling it up brought only marginal improvement.
It's even still making the same dumb errors as Opus 5.5, such as forgetting to pass the flag to run only the fast tests and waiting 1h, or randomly stopping to ask me if it should first implement A or B, when both are required.
Not a great story for their IPO..
Seems like a new focus is electronics design, which is both interesting and a little sad to me. Manually designing electronics are one of the things I found incredible comfort in doing; seeing AI emerging in this space gives me mixed feelings.
PS Astra and Opus were both good with schematics (and Fable), but Astra was notably worse at layout than Opus. Astra can still review layouts well, just not make them as well (n=1 though).
Reference: https://x.com/METR_Evals/status/2118412205662314935
Pretty crazy that Anthropic already has another model waiting in the wings that is significantly better than this.
You are reading what the external testers wrote about the earlier checkpoint
> the number of Fable resets you get depends on how long you’ve been a paid subscriber [...] Subscription length is measured from the date of your first paid bill. If you canceled and came back, the time since your first paid bill still counts.
> These resets expire on March 8, 2027 at 11:59 pm PT.
tl;dr if you joined when Opus 4.5 became a thing, you got 2 resets.
That means you can lift your Fable max 50% usage and you'll just end up burning through your quota faster. You won't have more tokens.
"Resets your Fable limit, and takes this week’s Fable use off your weekly limit."
