Top
Best
New

Posted by acossta 4 hours ago

Perplexity and Intelligence(openai.com)
355 points | 238 commentspage 5
qphe95 3 hours ago|
I wouldn't say perplexity is great metric because GPT-4 and o3 have similar perplexity on this chart but o3 was a massive step improvement. Pachocki has an incentive to push perplexity as one of the creators of the original LLM paradigm but I would say we have kind of moved on from that.
alextheparrot 3 hours ago||
Almost verbatim addressed in the post under “Perplexity as a shared metric has challenges”
namr2000 3 hours ago|||
This metric is purely scoring the ""intelligence"" of the base model, not including RLVR/chain of thought capabilities. It's widely believed that the pretrain determines the ceiling that RLVR can take you to.
jrflo 3 hours ago||
I think it can still be a useful metric, but probably isn't the "one true" metric. The trend is pretty clear, and the acceleration of capabilities since December 2025 definitely aligns with my subjective experience.

There are probably some other axes where o3 is clearly better than o1, but with how complex intelligence is no metric will be perfect.

am17an 2 hours ago||
A good and common metric. It will be interesting to see if other labs agree, but thank you for arriving at a metric and following up on the slowdown talk. I do hope others pay attention.
coolontheintern 2 hours ago||
Perplexity is definitely a useful metric for pretraining, but doesn’t mean much for post-training. In fact, low perplexity in a domain during pretraining doesn’t always translate to better behavior in post-training. Moreover, given that most of the recent progress and most of the frontier compute is going into post-training, I think it isn’t a very useful metric for intelligence.
londons_explore 2 hours ago||
Is this perplexity per-token or per character or per byte?

Also, is that graph assuming no changes in tokenizer if the measure is per-token?

amelius 2 hours ago||
Do we have a clear picture now when the ability to generalize is achieved?
hansvm 1 hour ago||
That's a nice story, but is there any evidence 2yo models with modern harnesses and the same compute expenditures wouldn't have produced the same results? This is written as some combination of a sales pitch and post-hoc rationalization without even bothering to control for the most basic confounding variables. Even if true, TFA isn't convincing, and not offering any more information than the conclusion it isn't helpful for any other purpose either.
andai 2 hours ago||
When we approach 0, are they gonna stop sounding cringe?

Actually, trick question I suppose -- the base models already don't sound cringe, it's post-training that adds that!

(Well, some of them do, because a lot of companies are putting synthetic data and LLM outputs into the pretraining phase (as well as bot-infused post-2020 common crawl, I suppose), but the early base models were fully cringe-free.)

TheArcane 2 hours ago||
> Publishing our internal perplexity measurements took some consideration; it gives competitors a clearer picture of where we stand,

very noble of you, _Open_ AI

ranyume 1 hour ago||
I find it convenient for labs that the proposed industry standard is a metric that only labs can calculate, using their own datasets that they won't share.

Clearly, if this should become an industry standard, public regulators must be able to audit both the datasets and how perplexity is calculated.

ericflo 1 hour ago|
I think "common limits based on perplexity" is a radical, offensive concept. A single clever individual could run afoul - you should have the right to be clever!
More comments...