Still I can'y help to wonder, are we hitting the wall in terms of frontier capabilities since Fable/Astra? Or is it this edge now, when all better models are kept private from the public?
It seems like they were saving these for just before ANT's IPO
Effort Model AA Index TerminalBench FrontierCode $/task
------ ------------- -------- ------------- ------------ ------
Medium GPT-6.2 Astra 57.0 58.6% 46.6% $1.62
Medium GPT-6.2 Sol 52.7 56.1% 41.4% $0.41
Medium Opus 5.5 48.2 52.5% 33.1% $2.45
High GPT-6.2 Astra 60.1 60.1% 47.1% $2.07
High GPT-6.2 Sol 54.9 58.6% 40.2% $0.56
High Opus 5.5 51.8 56.6% 35.9% $3.46
Xhigh GPT-6.2 Astra 61.6 64.1% 49.0% $2.52
Xhigh GPT-6.2 Sol 55.7 56.1% 42.8% $0.65
Xhigh Opus 5.5 57.4 59.6% 43.8% $6.05
But of course, benchmarks can't capture that "it factor."The influencers who get early access are using words like "worst", "terrible", "disappointing"
- https://x.com/danshipper/status/2122754729012777052
- https://x.com/mattshumer_/status/2122752192226406421
- https://x.com/theo/status/2122757647170703550>No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.
>I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.
>I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.
>It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.
I don't understand this at all. Is he just throwing different agents at vague open ended problems without any oversight and then getting upset when they don't do much?
I'm not exactly llm-pilled but surely even with a pretty mediocre model you could get something out of 300k spent.
Right now, most of the time I'm using 6.0 Astra for implementation work and the speed/quality is acceptable but at the moment Opus 5.5 is simply faster and better, but also more expensive. Things are really flipping around every 2 months or so, because at that time 5.8 was top dog without a doubt.