Top
Best
New

Posted by acossta 4 hours ago

Perplexity and Intelligence(openai.com)
390 points | 253 commentspage 8
f3408fh 3 hours ago|
Interesting that they work on weekends. Is it just because they're all basically millionaires living the dream so they don't care? Every place I've worked at has discouraged working late and weekends. Could also just be a scheduled post.
reasonableklout 1 hour ago|
The AI industry is in the most competitive race we’ve ever seen with trillions on the line and impacts on national security, of course they work weekends.
zurfer 3 hours ago||
The chart pretty much explains why I was freaking out about AI at GPT-4, because the trend of improvement has been so steep. And then followed a strong slow down, but now in 2026 things have really accelerated again. My hunch is really that coding is a freaking important input for AI and we might have actually entered some form of recursive self improvement. I know some people hate the term but I feel it's somewhat fine as coding is saturated (models can get better at it) and it has many levers for model training and inference (from algorithms to data).

Anyway, I think it's good idea. Would love to see all models on this chart, but honestly we might not be able to trust it going forward as it becomes a target.

rhdunn 3 hours ago|
Domains like code are easier to create autonomous RL loops for as you can easily check what the LLM generates by running it through a compiler, etc..

Domains like creative writing are harder as the automated grading suitable for autonomous RL is difficult (aside from spell checking and grammar checking).

bigmadshoe 3 hours ago||
> Given that the main way new surprising capabilities emerge is by progress on compression

Is it just me or does the article completely lack proof of this premise? I see a plot of perplexity dropping over time, with the implication that performance has risen as a result. This is just a correlation. What else did they change about post-training during this timeframe? This doesn't feel like a real experiment or real science.

joe_the_user 1 hour ago||
Quized ChatGPT about the article...

I had trouble seeing it in the article (it's hidden in footnote 3) but it seems perplexity/log-loss is tracked using the model's ability to track an unreleased code base of OpenAI.

The log-loss involved in code has declined more for code than prose in all of the models but has supposedly declined everywhere. [1]

At the same time, the decline of the latest models on a common sense benchmark might be meaningful: https://news.ycombinator.com/item?id=50588916

[1] https://openreview.net/pdf?id=SHMj84U5SH

fortran77 1 hour ago||
Perplexity goes back to Jelinek's IBM speech recognition group in 1977. Same Jelinek who supposedly said every time he fired a linguist, performance went up. This post is that quote with a bigger electric bill.
orbital-decay 3 hours ago||
Plain perplexity as a 2026 metric? It has to be marketing.
FergusArgyll 3 hours ago||
Interesting that he's attributing so much to the base model. I thought the majority of the gains are due to post training
Ohentis 3 hours ago|
If the majority of gains were due to post training, they just wouldn't do pre training. Many of the tasks people are trying to do with these llms (math and coding) don't actually benefit from natural language skills.
rfgplk 4 hours ago||
What's the perplexity of the average person?
kingstnap 4 hours ago||
Beating even a very small LLM at next token prediction is basically impossible.

[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2

We knew this many years ago. Like GPT 1 era.

Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.

Language modelling game https://rr-lm-game.herokuapp.com/

Ohentis 30 minutes ago|||
This game doesn't seem very fair to humans imo. I say this mostly in defense of LLMs. Beating humans would essentially imply overfitting but I don't think this is a very good show of that.
rfgplk 3 hours ago|||
Will try this out, thanks
jrflo 4 hours ago|||
Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain...
amluto 1 hour ago||
We are, though, but only partially. When listening and reading, we pretty clearly build a model of what we’re hearing or seeing and use it to correct for the rather noisy channels we use.

But when thinking, I’m fairly confident that our ability to predict the next few words of a book on a complex topic (in its words as opposed to the reader’s own words) is at best weakly correlated with intelligence or with subject-specific skill.

I certainly feel like I’ve learned a subject better when I can solve a problem differently from a solution I’ve seen before.

Kotlopou 3 hours ago|||
I linked a forum post asking this exact question elsethread (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...), and the answer is frustratingly dependent on the context length, the dataset, and the knowledge of the person about the topic of that dataset (e.g. I would have high perplexity in American history).
pishpash 3 hours ago|
I don't understand why anyone needs to exponentiate existing, well known concepts like entropy and cross-entropy to invent something called "perplexity". Somebody has Shannon envy syndrome.
More comments...