Top
Best
New

Posted by acossta 4 hours ago

Perplexity and Intelligence(openai.com)
380 points | 249 commentspage 7
hooloovoo_zoo 3 hours ago|
Hmm every annotation but the millennium one could be applied to every point....and possibly that one too lol.
fallingbananna 3 hours ago||
This is not the first time I've seen a chart of "Inteligence on a log scale" related to AI.

And it again feels equally dishonest/bullshitty.

You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.

It's just marketing talk. In reality all three are separate skills and the real intelligence requirements are, if you ask me, not that far away from each other.

To put this in an analogy, it feels like charting "athleticity on a log scale" and putting first running a marathon, then swimming across English channel, then benchpressing 200kg, each one presented as requiring an orther of magnitude being more athletic.

famouswaffles 3 hours ago|
>You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.

Where are you getting 'orders of magnitude'? and yeah I do think it's reasonable to say that Useful Programming requires more intelligence than the median human, and Solving millenium level math problems certainly requires considerably more intelligence than the median Programmer. What exactly is so absurd or unreasonable about that?

discreteevent 2 hours ago|||
> Useful Programming requires more intelligence than the median human

In the same way you could say that beating Gary Kasparov at chess requires more intelligence than the average human and yet we had an AI winter.

clutter55561 3 hours ago|||
The point is that talking, programming and solving math problems don’t lie on a single scale, and each problem requires different “skills.” Thus the comparison with running, swimming and weightlifting.
famouswaffles 2 hours ago||
We have long since known that there is a single factor that explains ~40% of the variation of cognitive tasks and abilities in humans (similar results can be found in other animals). We called that factor the general intelligence factor, or g.

Essentially, Humans who tend to have higher verbal skills also tend to have higher spatial skills, better memories, and faster processing speed. Someone with the cognitive ability to become an exceptional chemist might just as easily have become an exceptional mathematician or software engineer (all else being equal).

The knowledge and skills required are obviously different, but the underlying cognitive abilities that make someone successful in one intellectually demanding field often transfer to others.

Betelbuddy 3 hours ago||
This has a central argument around perplexity without ever properly defining it mathematically....

Then talks about cross-entropy but fails to explain the relationship between entropy, log-perplexity, and perplexity itself.

For an argument that relies so heavily on a quantitative metric, omitting such a fundamental explanation is poor.

programjames 3 hours ago|
The readers they care about already have this cached.
cuttothechase 2 hours ago||
where is Terra?

Is there a public link of all their models and where they fall along this or is this cherry picking points to fit the curve?

a3w 2 hours ago||
5.3 Codex was not a "useful coder", wat?
post-it 2 hours ago||
Claude, make a graph where the x axis is time and the y axis is a line heading toward something impressive, and then write me an article about it.
Leary 4 hours ago||
Looks like their latest internal model performs somewhat similarly to model 4 of anthropic.
artninja1988 3 hours ago|
How can you tell? Is there a chart that shows it on such a perplexity benchmark?
RAWGOD 3 hours ago||
Well, if you accidentally leak the evaluation set into the training set, you can reach perplexity 0.

Or the training set might just grow to be so big, defining a truly independent evaluation set is almost impossible.

eulgro 4 hours ago||
Didn't they hit a data wall roughly two years ago and had to rely on artifical datasets for pretraining i.e. LLM-generated?

I would expect the entropy of artificial datasets to be lower, thus easier to predict and scoring better on "perplexity"?

chubot 3 hours ago||
I thought that too, but models are trained on observation now, to increase agent performance. This is new data that is separate from Internet crawls. I think it's done with reinforcement learning with verifiable rewards

https://github.com/opendilab/awesome-RLVR

Examples:

- When playing Go or chess, nobody may have written or published web pages with a particularly clever end game, but an agent can use CPU time to discover these games themselves, and learn from them. Human Go players report learning strategy from what AlphaGo discovered.

- Agents use Unix shell and are trained with it. So they can discover things and combine things that nobody has written about on the Internet, like how to debug a rarely used driver, or an interaction between two drivers. And they can see patterns in all the drivers that humans may have a hard time even seeing, leaving aside the motivation to publish on the Internet.

This explains why the performance is so "jagged" - great on coding, math and closed games, but less good on things involving the real world. We probably did hit a data wall, but then we got new sources of data.

rhdunn 2 hours ago||
An interesting experiment would be to create a transformer model that is just designed to play Chess or Go. Specifically, the only thing that should change is the number of tokens, consisting of:

- the row of the move;

- the column of the move;

- the piece moved;

- any special/additional states from that move -- check, checkmate, castle, etc.;

- maybe an indication of whose turn it is (i.e. a black piece and a white piece token).

You could then train the base model on published game transcripts and use the interactive RL approach like AlphaZero and LeelaChess use, as well as what you mention for LLMs.

Why this is interesting is that it should give insights into how LLMs/transformers learn Chess/Go without having other data affecting the results. Specifically whether the token semantics identify valid moves, etc. and whether the context window encodes gameplay strategies across the different layers.

jrflo 4 hours ago|||
Yes, they started to hit a data wall, but I thought that much of the progress of the last couple years came from reinforcement learning after pretraining, which is mostly building complex environments for the models to reason in, rather than accumulating piles of raw text.
redox99 2 hours ago||
I'd hope they measured PPL on the same dataset for all those models. And one that's fair and doesn't benefit some models more than others (probably non synthetic dataset).
More comments...