Top
Best
New

Posted by jasondavies 58 minutes ago

GPT‑6.2 Astra and Sol(openai.com)
415 points | 212 comments
intenex 50 minutes ago|
Performance is insane for the cost if those benchmarks hold up. Looks like they did time this for right before Anthropic's IPO. Curious what Anthropic releases in response, and how these do on real world tasks.

Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?

mlinsey 14 minutes ago||
If you have the quota or budget for it, I recommend vibe-coding your own personal benchmark. I have a suite of 8 projects that I made with models this year, and replay them - give the new model a checkout of the repo at that point, and the project spec. For the design task, I have it throw up a blind voting page for me to quickly judge, and for implementation I have a review agent do a general score and also hunt for problems/edge cases that I really found.

It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.

(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)

grey-area 12 minutes ago|||
This is much more interesting IMO than any public benchmark. They are definitely targeting those benchmarks and model capability does seem to have plateaued somewhat.
Havoc 6 minutes ago||||
What type of tasks did you use for this? Evaluation?

Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes

srcreigh 9 minutes ago|||
please update !
xtracto 1 minute ago|||
>Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?

Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle

(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.

alastairr 42 minutes ago|||
I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.
pkthunder 38 minutes ago|||
We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess.
InsideOutSanta 18 minutes ago|||
5.6 is in no way representative, or comparable to 6.1. In fact, the 6.1 models are so good that I've just used Terra 6.1 xhigh for most tasks.
SOLAR_FIELDS 3 minutes ago|||
GPT models in my anecdotal experience are worse at drifting into the weeds on long running tasks, and continue to take what you ask way too literally. I have come to the conclusion that Anthropic’s models are just better at teasing out and intuiting what you actually mean even if you under specify it.

They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds

mi_lk 9 minutes ago|||
sol 6.1 is meh
zachthewf 25 minutes ago||||
why do you guys have to use 5.6?
Den_VR 14 minutes ago||
IYKYK. But in short because of a lack of alignment with product strategy
drdexebtjl 30 minutes ago||||
I have the exact opposite experience.
Doohickey-d 28 minutes ago||||
Well, yes, if you switch from current claude model to something that was released in July, of course it will be worse...
ok_dad 31 minutes ago||||
5.6 series is worse and more expensive than the 6 series. Try the latest Luna and sol; I use those for 100% of my work right now, with sol doing only 10-5 percent of it. I code in go and review the work and edit it extensively though, so I don’t know about how good those two are at vibing slop.
krzyk 22 minutes ago||
5.6 was better than 6. 6.1 is a bit better than 5.6 from my tests
throwitaway222 9 minutes ago|||
Bro you have to be at 6+. 5.6 is EARLY JULY
seizethecheese 37 minutes ago||||
Yeah, in real SWE tasks, 6.1 is bad.

We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)

(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)

OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.

locusofself 18 minutes ago|||
I've been building quite a complex desktop app (a digital audio workstation) using 6.1 Sol and it's been very good for me
slekker 21 minutes ago|||
Is this an ad?
seizethecheese 17 minutes ago||
No. I'm personally building agentduel.com, specifically with this problem in mind (benchmark performance not matching real world usage). It's generally accepted by HN community to make relevant comments about your own projects, as long as its not hamfisted.
Corence 21 minutes ago||||
Yeah, MaxxBench was shared here a while ago and showed most models think about how a grader will evaluate their work on DeepSWE in 95%+ of traces. Strong models definitely realize they're being graded and start to game their output.

https://maxxbench.unlap.ai/research/why-maxxbench/

skerit 36 minutes ago||||
I had the same impression. None of the GPT-6 models were as great as they claimed, they never surpassed Fable or Opus 5.5 on my use-cases.
suddenlybananas 39 minutes ago|||
You make it sound as though the models are like not working to their full potential instead of being a little bit overfit on the benchmarks.
revolvingthrow 34 minutes ago|||
> Looks like they did time this for right before Anthropic's IPO.

Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).

I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.

arscan 10 minutes ago||
My reaction as well - at the very least now is probably not the best time to knock down the realized valuation of a peer when they are effectively setting the market value of the whole category of ai companies, including yours… right?
the_duke 15 minutes ago|||
> Looks like they did time this for right before Anthropic's IPO.

While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.

These are incremental bumps, not big new models that have a huge impact.

6thbit 1 minute ago|||
They're going after anthropic's market through cheaper intelligence per token.

I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.

karimf 27 minutes ago|||
I'm very curious about this as well.

Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.

I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.

antupis 6 minutes ago|||
Why IPO if you have RSI and possible ASI?
esafak 5 minutes ago||
So investors can cash out on the time frame they promised investors.
ppierre 7 minutes ago|||
Yeah we see some Really Stupide Intervention from those who drank the Kool aid. IAS2[] start to be a real thing.

[] IA Stockholm Syndrome.

behnamoh 5 minutes ago|||
> Feels like we're way past the pelican benchmark being useful at this point.

It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.

ijustlovemath 45 minutes ago|||
I've been having fun with having it build interactive Three.js demos of devices, with setup screen, a few different operational modes, mechanical movements etc
ranyume 18 minutes ago||
You're looking for a vibe check?
Someone1234 25 minutes ago||
My big hope for this release is - Have they got their massive over-engineering under control?

Say what you will about Opus 5.5, but it has a good "gut instinct" for how large/complex to design the solution. The previous Astra just absolutely loved pointless abstractions, excessive numbers of files, and just over-complicating very simple things within an inch of their life.

And, yes, you can prompt your way out of anything. But needing a default prompt to tell OAIs models to work better isn't preferable, particularly when that sometimes gets lost in sub-agents and the over-engineering returns into parts of the work.

scandinavian 14 minutes ago||
I've been using both sol and astra to write some python and rust, and they generate the sloppiest code I've seen in a long time. I would be embarrassed to commit anything they write. The rust code especially was terrible. Had it implement a reader/writer for a binary format and it was just littered with magic numbers, terribly named variables, very compact code visually etc. It produces working code, so I guess the GPT models are mostly for people who let AI take the wheel 100%.
incrudible 17 seconds ago|||
I have limited experience with OpenAI models, but I have to say that Opus 5.5 has not improved on the terrible writing style, and I am struggling to prompt my way out of it. It always needs an extra pass dedicated just to rewrite all the prose.
holzhaker1 18 minutes ago|||
yeah i feel like with astra they went all in on 3D modelling which encourages "just one tweak, just one more tweak" which makes it detailed enough, but in coding that same behaviour just sucks.
uw_rob 13 minutes ago|||
Honestly I don't even know if you can prompt your way out of these types of behaviors - stylistic tics and judgement seem to be extremely difficult for the models to overcome. It seems we are tied to what the labs do unless we want to dive into and get dirty with models yourself. In retrospect with time it becomes obvious that the previous generation of models couldn't be prompt engineered to do what the current generation of models do. There's just no way you will prompt engineer Opus 4.8 to write as well as Opus 5.5. I'm guessing with time we'll see that there was no way you could prompt engineer Sol 5.1 to be as token efficient as Sol 5.2
contron 5 minutes ago||
I have found no way to overcome these kinds of things myself, even when showing the model examples of what not to do. Theres not really a good incentive for the labs to train them not to. If the model can overengineer, you spend more tokens. If you then prompt for the model to slim things down, thats more spend. If anything should come from the next generation of models, I hope its a sense of brevity and TIGHTNESS.
pants2 9 minutes ago||
Lol I made the mistake of having Astra design and build a complex system without any input (experimentally). Seemed functional at first and shipped it internally and it had dozens of complex bugs, so I started digging into the architecture and it was a monstrosity! Definitely 10X more complex than it needed to be.
simonw 28 minutes ago||
Pelicans for 6.2 Sol: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

And for 6.2 Astra: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Here's an updated comparison grid for the entire GPT-6 family: https://static.simonwillison.net/static/2026/openai-pelicans...

UPDATE: And if you're bored of pelicans, here's what I got for:

>

6.2 Astra (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...

6.2 Sol (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...

For comparison, here's Fable 5.2 Medium: https://tools.simonwillison.net/yonkers-rises and Gemini 4 Carbon: https://static.simonwillison.net/static/2026/yonkers-mech-st...

vopi 21 minutes ago||
This might be in other pelicans, but 6.2 Sol/Astra is the first time I'm noticing a model shade the leg further in the frame darker to mimic depth (or a shadow).

It's also interesting you can clearly tell 6.2 Astra does a much better job of the above. 6.2 Sol attempts it at higher reasoning levels, but it's crude in comparison.

I wonder if it comes from the (seemingly) recent inclusion of 3D environments in training data.

RGS1811 10 minutes ago|||
The emergence of the beach setting and the scarf is interesting.
I_Write_It 20 minutes ago|||
They all seem very good, with a lot of attention in detail ! Thx for the Pelicans!
adverbly 18 minutes ago|||
What's with everyone saying these Pelicans are great?

These are some of the worst regressions that I've seen in a while.

Pay close attention to the stacking layers around the feet, pedals (sometimes there aren't even two pedals), wheel, and two legs. Some pretty obvious mistakes in most if not all of these...

Edit: And what's up with the single arm on the two highest Astra images?

edit2: really? I'm getting downvoted because people don't like that I pointed out a mistake in a Pelican svg? Is that where we're at?

svachalek 4 minutes ago||
The highest Astra images give the pelicans wings instead of arms, which does seem like a valid interpretation.
firemelt 23 minutes ago|||
any index link for all ur pelicans i want to see fable and opus
tintor 23 minutes ago|||
Sol 6.2 max is still failing on pelican benchmark. Left leg is jammed in chain.

Astra 6.2 max did a similar mistake on the left bicycle pedal.

frangonf 13 minutes ago||
But Astra 6.2 low and medium didn't make this mistake.
BoldColdHold 22 minutes ago|||
Wow the yellow is leaking into their llms too.
Hacktrick 25 minutes ago|||
I'm not sure if this is a useful benchmark any longer
p11p 26 minutes ago|||
Are you getting any signal out of this anymore? All I can see is "good pelican" and "dogshit pelican". These ones look pretty good.
EugeneOZ 23 minutes ago|||
I do. 6.2 Sol xhigh and max show the results comparable to Astra 6 max, and Astra 6.2 only shows any improvement on "max" reasoning effort level.
adverbly 20 minutes ago|||
You're kidding right?

These are some of the worst pelicans I've seen in a while.

Pay attention to the stacking layers near the legs. Sol literally never got it right, and Astra regressed on it multiple times...

Also what is that bending bar from Astra?

EugeneOZ 24 minutes ago||
still the best benchmark! thanks
Topfi 42 minutes ago||
Office work has never been OpenAIs strength, no shame in that by itself, but these Slideshow examples look particularly awful, far worse than what I myself got out of GPT-5.6 Sol about two weeks ago in a single prompt.

Same for Workbooks and memos, those examples read inferior to some pre Opus 4.5 and Kimi K2 Thinking docs I have kept around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.

I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.

Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5. GPT-5 to GPT-5.4 were truly groundbreaking, but since then there have been regressions in what made those models so useful and reliable.

Edit after the first, incredible simple test prompt using GPT-6.2 Astra on High using Codex, this was the response to me sharing a screenshot of the very much not in scope result and asking how that happened:

"[...] I added controls and reconstructed interface chrome that weren’t authorized [...] visuals often failed to show the feature [...]"

Great, so it's one of those releases again. Will still run the full evals of course, maybe I'll get surprised, but not a good initial experience. Please OpenAI, get it together, we need actual competition.

EugeneOZ 19 minutes ago||
I see the work of Claude Slides every week, and when I saw slides in that article I was like "...omg, what a disaster... And they are bragging??"
Topfi 11 minutes ago||
The odd thing is that while there is chasm between 6.1 Sol and e.g. Sonnet 5.5 in the seconds favor in such tasks, it is not this bad irl. I really don’t know how they managed to seemingly nerf their own models in an announcement showcase.
bluepeter 27 minutes ago||
For sure. Feels like an episode of Silicon Valley (or even Office Space) and their lead actor looks a lot like TJ Miller.
Topfi 16 minutes ago||
Makes one wonder what samas handwriting analysis says…
theanonymousone 50 minutes ago||
Happy for the new Sol. It's my workhorse on company account.

But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.

ariwilson 6 minutes ago||
Why not Haiku 5.5? It's substantially smarter than Luna (though more expensive - still cheap though).

(personal benchmark) https://fairbake.dev/signature/agents/haiku-5-5 vs https://fairbake.dev/signature/agents/gpt-6-luna

I've found Luna to be very lazy as well (claiming to have run tests/builds when it hasn't).

esafak 19 seconds ago||
That's why it's so token efficient. It is so lazy that I feel like it's taunting me to do the grunt work. I wonder if there is a benchmark for that.
SomeonesAccount 47 minutes ago|||
I find that Luna tends to misinterpret instructions in READMEs and Agents.mds. Maybe it is something specific about my setup, but I found that Terra is the lowest I can comfortably use on its own (Luna can be driven by larger models in the form of a subagent and it works great for that)
MrBuddyCasino 41 minutes ago||
I used Luna Max a lot, until Deepseek 1017. Then never again.
jrflo 22 minutes ago||
Artificial Analysis benchmarks just updated, both Astra and Sol are on the Pareto frontier in intelligence/cost per task:

https://artificialanalysis.ai/models?speed=output-speed-over...

6thbit 6 minutes ago|
Seems that OpenAI's strategy for Anthropic's IPO week is cost pressure.

Rather than trying to release a better model than fable/opus, they're just making them look extremely expensive. Let's see if Anthropic reacts here in any way.

freecodeio 3 minutes ago||
to quote a comment about claude:

the only impressive thing about claude is seeing how much it costs

underdeserver 49 minutes ago||
I wonder why they didn't show Gemini 4 Carbon's scores on the science benchmarks, especially since Google seems to be pushing it for that -

https://x.com/pushmeet/status/2120814064863461440?s=20

(FD: Google employee, I don't work on Gemini)

tedsanders 21 minutes ago||
I work at OpenAI and I made the plots. Gemini 4 Carbon was omitted when we couldn’t find public data, usually cost. No intent to hide anything.

If we missed anything, feel free to point me to it and we can add it.

underdeserver 8 minutes ago||
So all the plots are based off public data? You don't run the models internally yourselves?

(Again speaking only for myself)

holzhaker1 17 minutes ago|||
they are probably trying to authenticate their phone number to access agy
eknkc 23 minutes ago|||
Probably still waiting for it to finish reasoning on the benchmark. Should be ready in a week or two.
knuppar 45 minutes ago|||
Yeah it feels shady how Carbon was selectively plotted (or not), unlike Opus/Fable
TiredOfLife 46 minutes ago||
Two posibilities. 1. They couldn't figure out how to actually use those Google models.

2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.

ilaksh 11 minutes ago||
Something like 2/3 of the comments in this post when I started reading were reviews of previous GPT models stated almost as if they were important information for evaluating the new models. I feel that when a new model comes out, it would be more helpful to refrain from leaving a review until you have actually done some minimal testing of the model.
antupis 2 minutes ago|
But it usually makes sense new models probably use somewhat same training pipelines and data.
joenot443 42 minutes ago||
The video is pretty funny. It's like a callback to 90s/00s office life?

I'm immediately reminded of the famous Excel elevator commercial. [1] Perhaps that was intended?

[1] https://www.youtube.com/watch?v=HegoJaXOIqQ

pmg102 6 minutes ago||
Yeah it's 100% a nostalgia callback to this commercial, right?
apetresc 34 minutes ago|||
I guess it's kind of funny in a tacky way, but pretty much every scenario is just "oh shit I was too lazy to do my job and now I'm about to be called out, better throw it into ChatGPT literally ten seconds before I get in trouble."

Not sure this is the way OpenAI should try to make their case for why AI is a net societal benefit. It feels more like an SNL skit making fun of an OpenAI commercial.

Like, why not just throw in a high school student cheating on a test with their phone under the desk, at that point? Oh but it's a Nokia flip phone, haha, just a joke.

ilaksh 22 minutes ago|||
What do you think would be a non-tacky and effective marketing approach targeting office workers?
black_knight 26 minutes ago||||
Feels like this is exactly what people do in meetings I am in these days. "Claude give me this…" is the start of a lot people's contributions to the meeting. It is then followed by the most inane, over-designed artifact with no real content.
ceroxylon 17 minutes ago|||
It is bizarre and bordering on offensive, if an employee throws some last minute slop at me, I would wonder why I am enabling a meat proxy and not prompting the LLM myself.

At least the ad from the 90s shows that this is nothing new; the important, time-critical task being demoed is... a basic spreadsheet expensing "exotic excursions" for some C-suite so they know how much they spent on safari???

noir_lord 31 minutes ago||
Funny but Dystopian, the message is "Half arse your job, the clanker has you".
astro-lizard 16 minutes ago||
the message is definitely intentional, the goal is to de-skill office workers.
jorl17 27 minutes ago|
I really want GPT models to be good, but Sol models are so slow I cannot use them. Maybe the 500$ subscription (or enterprise packages) are better, but it is unbearably slow. I've tried the fast variant and low reasoning levels and there isn't a significant improvement.

Couple that with the lack of reasoning traces, and I cannot even have a clue what the model is doing while it is working. This is perfectly fine for when we are developing with care for the code (in which case I will know what it is doing by reviewing the output), but not for purely vibe-coding scenarios. And the thing is....if I care about the code...I'd rather trust Opus or Fable.

ativzzz 21 minutes ago|
Does speed really matter? Most of the time I get the AI to do something, I immediately switch to another one of my things I have in progress and then come back at some point later. I'm never sitting there waiting for it to finish
treis 13 minutes ago|||
It'd be a lot better if you didn't have to switch. It's not nearly there for coding tasks but for other things instant responses are a game changer.
edvardas 15 minutes ago||||
Personally, my cognitive RAM is limited, so I'd rather work on a few tasks at a time.

The speed is not a big deal - Sol works just fine, but I find it a lot more fun to work with Opus due to its speed

Pedro_Ribeiro 15 minutes ago||||
It depends on the task. For frontend work, for example, it's table stakes.
jorl17 11 minutes ago|||
It depends on the project and task, but for the kinds of projects I'm throwing at it, it matters quite a lot. I have often been focused on building just one project, without the possibility of many parallel tasks due to machine contention (and unfortunately offloading to the cloud is not an option). This means all tokens go to the same project, and they all go sloowwwwwly.

The difference is quite staggering. I swap Anthropic and OpenAI models around when tokens run out, and I can do 10x the amount of work on Anthropic's models. Plus, like I said, I can periodically sort of see what it is thinking. It doesn't just say "Recent potato tests caught an off-by-apples error. Regression tests are being written and the gate will run again. I will not call this ready until tests reach expected coverage" every 5 minutes. Sure, it spits claudese, but I'd rather have claudese to parse than this OpenAI nothing-burger of a report.

More comments...