Top
Best
New

Posted by acossta 4 hours ago

Perplexity and Intelligence(openai.com)
340 points | 222 commentspage 3
armcat 1 hour ago|
It's absolutely insane when you look at the period between GPT 3.5 (March 2022) and GPT 5.2 (December 2025) - more than 3.5 years, and then just look what happened this year - the rapid improvements almost on a monthly basis, and log-perplexity just taking a dive (conversely, "intelligence" skyrocketing). I'm wondering if December 2025 will be known as the beginning of something?
criddell 56 minutes ago|
When I think about the people I consider very intelligent, I think it's their ability to learn that impresses me, not memorization of facts (although that's a part of learning).

Why are the major AIs so terrible at learning? It seems like things like self-driving cars could be solved over a long weekend (about as long as it takes to teach a kid the basics of driving) with a great learning model.

MrCheeze 3 hours ago||
I always had the impression that model capabilities were stagnating around GPT-5.0, and this finally broke around the release of Claude 4.5 later that year. Seems like their own graph corroborates that things were pretty stagnant then. I wonder what changed?
kalcode 3 hours ago||
Oh oh! The models actually know what happen as well if you want to dig deeper. Pre-scale brute force training hit there limits. They broke through late 2025 due to RL training techniques that taught them to fail, fail, fail to succeed instead of spin off into a corner with nonsense.

Basically we started tuning them to be smarter and able to handle 2 to 4 hour tasks.

I believe this will hit it's limits as well. When IDK.

SwellJoe 2 hours ago|||
Given the current models from Anthropic and especially OpenAI have a hard time constructing coherent English sentences that a human can understand even while being better at code and math, I'd say they're reaching that limit; can't use a model that you can't communicate with. The Deepmind folks seem to have figured something out on that front with Neon, though, as it's both smart/capable and a clear communicator.
bscphil 1 hour ago||
I think they're probably just RLVRed into oblivion to do well at coding, because that's what 98% of their revenue comes from. That's how you get the precise language which still feels awkward and stilted, because internally the model learns to align specific peculiar expressions to precise meanings, which are useful when you're trying to write code. There's no point in using a complete sentence when you're mostly talking to yourself; the result is intelligible, just terse to the point of absurdity. (And some models, including Google's, still have a strong tendency toward flattery thanks to RLHF. Ah well...)

Just to have a shared example of AI text, this is pretty instructive I think: https://statmodeling.stat.columbia.edu/2026/11/21/by-program...

Where a human would write "by the end of the program" the model (feels like Claude to me but idk) writes "by program end". Where a human would write "There are two scheduling considerations for program hosts." the models writes "Two timing windows worth flagging."

The latter text, "two timing windows worth flagging" is highly specific and precise if you're a model keeping track of something in the docs of a repository across contexts, but it's terrible human communication.

ACCount39 2 hours ago||||
This graph specifically tries to measure pre-train perplexity. It measures the "foundation" of a given base model - before RL is applied to it.

On long term tasks, "CoT + RL" of o1/o3 beats GPT-4.5 resoundingly. But on raw base model perplexity, GPT-4.5 is a significantly better model than any of the o-series.

The curve following GPT-4.5 and o-series suggests that newer models take advantage of BOTH better base model capabilities (perplexity on held-out data) and better RL.

kalcode 1 hour ago||
Ohh, neat thanks for the spotlighting that, I didn't know that.
famouswaffles 2 hours ago|||
They are reporting perpexlity on pre-trains not RL-tuned models.
chubot 3 hours ago|||
I'm not an expert, but I imagine it's what I wrote in a sibling comment -- training to improve agent performance (reinforcement learning with verifiable rewards).

That generates new data that's not on the Internet -- more true observations to learn from, compress, and generalize.

I think that improves the models in general, not just for agents. (And also I think it's the cause of the "Claudish" prose dialect we all started noticing)

I think that matches the timing, though I appreciate any closer analysis

jordanb 2 hours ago|||
I think most of the progress in the last few years was more RF and harnesses, especially wrt (for example) claude code.
Wowfunhappy 3 hours ago||
[dead]
andai 2 hours ago||
The divisions they drew in the log graph are of course arbitrary, but GPT-4 (the original 8k context version from early 2023) was able to unminify JS, as well as produce an autonomous coding agent. (This was via ChatGPT, so I copy-pasted the code into files, but it both designed and implemented it.)

So "useful coder" arrived about three years earlier, by my count.

nico 2 hours ago||
> Detachment from actionable evaluations. Perplexity does not directly correspond to a task we care about

The post makes a good case that at frontier scale this doesn't matter much, because perplexity gains turn into new capabilities. However, this has been an issue for me when training local models

Some agents wanted to optimize for perplexity, and we did all kinds of experiments, but at least for what I wanted (a small conversational chat model that can run in my browser, ~8M params), that was the wrong way to go. I could get excellent ppl numbers, yet still almost no coherence

What worked best, in my case, was to get better data, specifically, hand-crafted examples for the tasks I wanted the model to be good at. I'm guessing that perplexity might track capability when the model is big enough to turn the gains into general skills, but at a few million params it mostly buys you better-sounding word statistics

GodelNumbering 1 hour ago||
If you take the argument at a face value and follow, this also implies that any advances are largely correlated with base model sizes (note the latest drop is from Astra, not sol 6x). And importantly, to drop the log perplexity incrementally, you have to increase the base model sizes continuously and likely disproportionately.

In other words, there is only so far you can go with the model of a given size before needing to increase it for smaller incremental gains.

djeastm 2 hours ago||
So the jump in intelligence was like a kid that keeps getting better at reading and then one day they suddenly can do word problems. Now the worry is the student will soon be writing the textbooks and we won't be able to check the work?
matteoraso 2 hours ago||
Wouldn't reducing the entropy too much lead to lack of diversity of thought? An idea that I've had for a while is that LLMs are held back by having the same mind, leading to modal collapse. Ideally, an AI swarm would have different approaches to a problem so they don't all end up trying the same thing.
dsign 2 hours ago|
Drugs! Give'em psychedelics, let 'em see da colors!

Nah, just kidding.

The labs are training their models on the previous' version conversations, but I'm sure not all those conversations are weighted equal. So there are some conversations that are more "sparky", where both LLM and human work better towards achieving a solution. The next LLM iteration learns from them. In that sense, they don't need to hallucinate, but to do exactly what LLM dissenters say they do: "they just regurgitate something they saw (trained on) somewhere". To us humans, who haven't read all the books nor experienced a dozen million successfully completed tasks, that "rehashing of knowledge" (I prefer "recombination of knowledge") is more than good enough to solve ever more challenging problems...at least as long as there's a human in the loop with ever bigger ambitions.

Kotlopou 3 hours ago||
Oh, I always wanted to know how the entropy of English was looking! The human baseline is both surprisingly unclear and surprisingly better than even today's frontier models (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...). Though this is an extrapolation from 6 years ago, so take it with a grain of salt.

There's also the Hilberg model where the entropy rate keeps falling towards zero as you consider longer and longer texts, and at least here (https://www.youtube.com/watch?v=5eqRuVp65eY) data from LLMs showed no floor. It would be unintuitive if it turned out that you can compress sensible English text to, say, O(sqrt N) length!

WASDx 1 hour ago||
Do they say that they measure perplexity at the pretraining phase? I thought most of the advancements have come from RL during posttraining. Would an older model with modern posttraining have the same perplexity while being significantly better? Then I don't see it as a meaningful metric.
reasonableklout 1 hour ago||
A common view is that RL is eliciting behaviors learned in pretraining [1], so even though many recent gains are achieved via posttraining, pretraining is still extremely important because base models provide all the latent potential. This is discussed a bit in the OP.

[1]: https://www.interconnects.ai/p/elicitation-theory-of-post-tr...

famouswaffles 1 hour ago||
They are measuring the perplexity of pre-trains.
deadlypointer 1 hour ago|
Kind of expected, they are increasing number of parameters, and the model can fit the data better. Does not mean anything about "emergent capabilities" in my reading.
More comments...