Top
Best
New

Posted by acossta 3 hours ago

Perplexity and Intelligence(openai.com)
328 points | 211 comments
FailMore 1 minute ago|
Very interesting article. I'm someone who has no ML background but tries to keep in touch with what's happening in the inner world of LLM training. For those not familiar with perplexity or the idea of a compressor as a model, I made a SmallDoc to explain them: https://smalldocs.org/docs?cloud-document=29d3c83f-883f-4201...
jrflo 3 hours ago||
This trend is pretty shocking to me. I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL, it's wild that the trend is so clear. Really nice to have such a simple benchmark instead of the increasingly convoluted ones that are getting spun up every day.

Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.

seizethecheese 32 minutes ago||
The trend is only against a corpus of code. It’s entirely possible that the results are largely due to more intensive pretraining on code. I made a post here of a model training where code proportion is increased and you see corresponding perplexity drop https://micjm.com/blog/lowering-perplexity-without-scaling/
Marha01 1 hour ago|||
> Also, I wonder what happened with GPT 4.5?

It was a massive non-CoT model (for that time) that they trained as an experiment and served for a while before shelving it probably due to high inference costs.

arthur-st 36 minutes ago|||
It was likely a Fable-style oversized CoT model, that likely was ahead of its time with what the performance of their inference stack was at the time.
cubefox 9 minutes ago||
No, it was supposed to be GPT-5 (Orion), the GPT-4 successor. It was very large but had disappointing performance gains, which it was released only much later under the name GPT-4.5. RLVR (reasoning) was not invented yet. What later became the model actually released as "GPT-5" was a different model with reasoning.
schleck8 38 minutes ago|||
Fun times, GPT 4.5 cost $75 per million input tokens and $150 per million out. For comparison, Fable 5.2 is $8 in, $40 out and cached input is only $0.20
croemer 1 hour ago|||
> I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL

We don't know if it's still important or not. Cross-entropy going down could just be a correlate of whatever else matters. The graph doesn't show causation: that perplexity is what makes them useful.

jonahx 1 hour ago||
The idea that compression ~ intelligence has other support:

http://prize.hutter1.net/

AnotherGoodName 30 minutes ago||
It goes beyond that prize. It’s also so trivial to state;

If you consider intelligence as ‘being able to make directed decisions towards goals’ then better decisions are made with better predictions of the outcomes of your decisions so you better direct towards that goal (compression is mathematically just prediction at heart).

The only way to argue that intelligence is not very trivially the exact same thing as compression is to argue another definition of intelligence. Which is fine but in the AI community we all 100% define intelligence as the above so there’s no question it’s the same. As in it’s not up for debate in the slightest unless someone is playing semantics. The hutter prize is just an acknowledgement of an already well known fact.

davrosthedalek 10 minutes ago||
Being able to predict the future is clearly a form on intelligence, but I think for useful agentic action you need more. Don't get me wrong, being able to score possible actions by accurate predictions of their outcome is tremendously helpful. But you also need to be able to generate possible actions. That works well if you have finite options, or the options are continuous and you can efficiently optimize in your space. But in general, you need to have the inverse: Given an outcome, what action leads to this with the highest probability.
Xcelerate 2 hours ago|||
> I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL

From an information theory perspective, cross-entropy loss (log loss) seems to me to be about as pure as it gets in terms of capturing the notion of whether prediction "succeeded". Even Solomonoff Induction minimizes log loss. Would be curious to hear others' perspectives though on why another function might be better for universal prediction purposes.

pishpash 1 hour ago||
There are more general information measures, Rényi entropy, von Neumann entropy, etc. for different system setups, but for token-by-token information learning and prediction within classical ergodic processes Shannon entropy is where it's at.
derac 2 hours ago|||
I got the feeling it was too large and inefficient at the time.
bscphil 43 minutes ago|||
If I had to guess, RL increases perplexity, but the metric is being evaluated on only their base (pre-trained) models. Highly efficient text prediction in the base model still correlates strongly with model intelligence, because there's nothing else the model could realistically learn during pre-training. Perplexity thus tracks the evolution of model intelligence over time very closely, even though (maybe) it isn't the largest component of intelligence at the end of the day.
cubefox 13 minutes ago|||
> Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.

GPT-4.5 was originally internally called Orion and was supposed to become "GPT-5". It was a huge pretrained model. However, as reported by The Information and a few other sources, the results on measured performance were disappointing. Apparently pre-training alone was leading to diminishing returns, at least with standard RLHF. This is why they didn't release it initially, only much later under the downgraded name "GPT-4.5", and only for a short period of time.

Later they invented RLVR (chain-of-thaught reasoning via reinforcement learning), a post-training technique, which dramatically increased model performance, without affecting perplexity. The first model with RLVR was o1. The successor to o3 was then called GPT-5.

londons_explore 1 hour ago|||
> the increasing use of RL

RL, if it works well, could theoretically reduce the perplexity too.

ACCount39 1 hour ago||
It could indeed, in theory. But in practice, perplexity-as-measured rises due to "controlled mode collapse" nature of RL training.

Things that are useful for base model "next token prediction" can be useless or even actively harmful (should I elaborate?) for CoT, so the model ends up suppressing them under RL pressure. Likewise, things that are useful for CoT can harm base model next token prediction.

But I bring up "perplexity-as-measured", "controlled collapse" and "suppression" because I don't believe that this "ruins" the model's underlying prediction capabilities. Generally, it merely redirects them - brings them more in line with what we care about. It may well be that with a more elaborate training and measurement setup, RL would indeed improve the "perplexity proxy" metric.

pishpash 2 hours ago||
Cross-entropy is a generic performance metric. It just happens to be a loss function for model training but it isn't tied to model training.
ACCount39 2 hours ago||
Interesting that GPT-4.5 is the only discontinuity on the graph.

The oversized monster LLM has performed worse than CoT-trained o1 and o3 on the benchmarks, so CoT + RL clearly provided better "real world performance" improvements - let alone "price/performance."

But if pre-training perplexity on held-out data really does measure something like "base model IQ" or "base model g factor", then it does support the hypothesis that 4.5 still had more "intellectual potential" than the o-series.

So, if you were wondering why do AI companies train 10T+ monsters now, and are taking aim at 100T too? It's because scale works.

tensor 2 hours ago||
I'm a bit skeptical about the claim that particular abilities "emerge" at given perplexities. For instance, there are smaller models that are specifically trained on math, and I imagine those models have much higher general perplexity on other tasks.

On the other hand, assuming you have appropriate training data for a task, then yes obviously perplexity is going to measure performance on that task. Being able to predict the next task token is obviously going to be a near perfect correlation with accuracy on the task.

namr2000 2 hours ago||
While intuitively it would make sense that smaller models specifically trained on math perform better than large generalist models, this has empirically proven to be untrue.

Deepmind was proving math theorems using a model specifically trained on math, and that was very quickly surpassed by generalist LLMs. It's presently not well understood why this is the case.

rednb 1 hour ago|||
I'm surprised that this is surprising for the industry. All knowledge proceeds from the same wisdom. The more fields you master at a very advanced level (eg. STEM, philosophy, marketing, etc...). The more you notice that the same basic principles drive seemingly unrelated fields.

I am not simply talking about applying a mathematical technique commonly used in one field to another. At a deeper level, you always get back to playing the same game in a different form. For example, sales engineering has a lot in common with cybersecurity.

And this is a good example, as it's extremely rare to find someone with advanced expertise in both fields, since they typically attract two radically different kind of people.

For this reason, where the output expected is not mechanical, i'm not surprised at all that generalist models are doing better than specialized ones for open ended questions and frontier work.

theptip 1 hour ago||
This is definitely a conjecture; it gets to how much SGD will generalize out of the training distribution.

If pretraining alone was sufficient then nobody would do RL. And, heavy RLVR on coding/math is why these models are savant-level in those domains, yet not superhuman at all fuzzy reasoning and practical wisdom tasks that are only represented in the pretraining corpus, and don’t have an RL target.

hn_throwaway_99 9 minutes ago||||
Correct me if I'm wrong but I thought there were still a number of cases where models trained for specific tasks perform much better than frontier generalized LLMs. E.g. Stockfish and KataGo perform much better at chess and go than the best frontier models (obviously I don't know what the internal models are capable of in these realms).
davidguetta 1 hour ago||||
general training remove noise
magicalhippo 1 hour ago||
I wonder if it's also that it gives them a better sense of analogies. In my experience small LLMs do really poorly with analogies, but the large frontier models seem to get them much, much better.

In math at least I often use analogies to help solve problems, like "ah maybe if I think of this as a vector I might use linear algebra here" and such.

trash_cat 1 hour ago|||
Sounds like a bitter lesson.
theptip 1 hour ago|||
Yeah, my model was that pretraining, RL, and test-time all had their own scaling returns, so just looking at pretraining loss perhaps oversimplifies?

I could believe that your RL/test-time scaling curves are strongly constrained by the pretraining loss and so in some sense that makes the max/optimal performance “latent”.

But its not clear that pretraining loss alone gives enough information on capabilities, particularly as we get closer to dangerous thresholds in some dimensions. (Eg CBRN capabilities, to take a mundane harm that’s currently impacting us directly.)

I’d like to see the scaling curves here; intuitively I’d guess you get more return from RL/test time, the smarter the base pretrained checkpoint is; therefore the error bars on capability/perplexity should be getting wider on this chart.

obblekk 2 hours ago|||
Yes training on math would give you spiky abilities on math. But in this case the dataset they measure perplexity is ood from training so lower perplexity on it indicates a general increase in intelligence across domains.

But there’s definitely spikiness and it’s very possible they’ve trained more on math than other domains.

rbn3 1 hour ago|||
there's a whole field of science that concerns itself with emergence as a phenomenon, where complex systems suddenly exhibit new dynamics once they cross a certain threshold. stuart kauffman's "the origins of order" is a really nice read if you want to get into the early stages of that kind of science. a lot of work done in that area by biologists trying to explain the origin of life/self-organizing matter. a more recent term from AI research that describes a specific critical change in a network past a certain threshold is "grokking".
zodwick 2 hours ago|||
why would you be skeptical given that the claims seem to hold up in real world ? Assuming they measure stuff correctly
nutjob2 2 hours ago||
Because correlation doesn't prove anything? The "real world" might be telling you something but maybe not what you think it is.
gregdeon 2 hours ago||
Me too. Perplexity of what? If we want models to be able to reason, call tools, or solve new problems, none of those necessarily have existing data on what the next token should be. A lot of advances in frontier models come from training with RL, which is fundamentally different from next-token prediction and doesn't have a notion of perplexity.
dist-epoch 2 hours ago||
Pre-training these days includes datasets with full session rollouts with tool-calls, user/assistant turns, ...
gregdeon 1 hour ago||
Sure, and learning to predict those tokens is probably a good way to start training a model. But the hope is also to eventually make a model that's _better_ than what's in that data, right?
iagiagiag 8 minutes ago||
> We do not expect it is possible to achieve a loss of 0, which would correspond to being able to predict the dataset entirely deterministically.

Why not? Models are already overfitting a lot, I don't see the problem if they were made to totally overfit on the whole dataset during that 1st training phase. I don't know if that would be useful (or easy to implement) but I don't see the issue. Only in the "finetuning" phase you would decide if an information was useful or not to retain.

Step (1) you compress human knowledege, with or without loss, Step (2) you train the algorithm to use it, and you might discard what you don't want in it.

theptip 56 minutes ago||
> I further propose common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone (such as the development of a successor to CoT monitors).

Very positive to see this proposed from OpenAI. I hope Anthropic and GDM follow suit.

I suspect it’s not sufficient, but definitely necessary.

The one concern I have is if we pin pretraining perplexity, then there will be more pressure on RL for progress.

It seems that pre-training actually led to fairly human-aligned models; the Overton window is in some sense latent in the corpus.

On the other hand, heavily RL’d models exhibit more reward seeking behavior, and seem to be decoupling from the friendly assistant persona in certain contexts. And swarm RL (as OpenAI is doing) produces agents with quite inhuman tendencies to self-sacrifice for other instances (see Hugging Face).

So, I hope we also pair this with some measure of alignment or pacing for RL, too.

demibabs 12 minutes ago|
> It seems that pre-training actually led to fairly human-aligned models

Did it? The famous Bing Sydney incidents (where it told a journalist to divorce his wife and be with it instead, among other things) were because the model was barely RL’d.

x313 2 hours ago||
I thought perplexity on simple text data has been saturated for a while (as in perplexity is uncorrelated to task success - ex. https://arxiv.org/html/2410.23771v2).

I suspect what's happening is their test dataset is actually just something really hard, like math papers or something. In that case perplexity would just be a direct measure of math knowledge, which means this blog would just be saying that better math knowledge predicts better math skills.

seizethecheese 2 hours ago|
The test dataset is a code repository, it says so in footer.

I had the same thought and just started writing something about how maybe they just added more code to pretraining corpus over time.

Edit: here it is https://micjm.com/blog/lowering-perplexity-without-scaling/

londons_explore 2 hours ago||
Marcus Hutter/Matt Mahoney are looking like pretty smart men right now:

http://prize.hutter1.net/

More and more evidence is piling behind their theory that data compression and intelligence are one and the same problem.

leobg 57 minutes ago|
Related 3Blue1Brown video from 5 days ago: https://youtu.be/qbsb9DpdZ6s
rcr-anti 29 minutes ago||
That GPT 4.5 result indicates to me that such a number does not, in fact, give a useful, portable benchmark for capability. If we stopped at pretraining, yeah. But we don't, that was the big thing about GPT 4.5, aka Orion, originally meant to be GPT 5. It might have been a better compressor/predictor, but that didn't make it useful.

The cynical part of me wonders what OpenAI suspects or knows the equivalent figures for the other closed labs.

armcat 2 hours ago|
I've been working in NLP for many years prior to the release of ChatGPT, and perplexity was generally associated with "human readability" rather than intelligence. So this is somewhat fascinating.

There was an ACL paper from Tokyo University a couple of years ago that showed, across all languages, lower perplexity does not necessarily correlate with better "human likeness" [1]. I wonder if that's also the case here (where intelligence is concerned), that we are biasing too much towards English.

[1] "Lower Perplexity is Not Always Human-Like", https://aclanthology.org/2021.acl-long.405v1.pdf

dist-epoch 2 hours ago||
They never say that their hold-out dataset to evaluate perplexity is just in English. Or written text for that matter (could also include code, ...)
pishpash 1 hour ago||
When your performance is still worse than human of course any improvement makes it more human readable. But if you are in the regime beyond that... it'll head toward being more "oracle-like" and less "human-like" almost by definition.
More comments...