Top
Best
New

Posted by acossta 4 hours ago

Perplexity and Intelligence(openai.com)
371 points | 241 commentspage 6
andsoitis 1 hour ago|
I’ve always thought the French AI company Perplexity chose a very poor name. It is perplexing why you’d pick a name with negative connotations.
thehamkercat 3 hours ago||
OpenAI hasn't released any competent model since opus 5.5 came out, i wonder if these labs are deciding to pass the "best model" card every quarter to each other for good public perception of all labs or something, because OpenAI can't be this incompetent
distantsounds 36 minutes ago||
Oh so intelligent yet they still struggle to successfully shoehorn this into everyone's every day lives.

Almost as if this stuff has extremely little practical use outside of niche categories.

Retro_Dev 3 hours ago||
Saying that perplexity is a metric of intelligence is misleading. Inkling-2-Mini was released recently, and I've measured its perplexity at 5.7 on a small varied text dataset. For reference, the graph OpenAI is glazing is plotting log(PPX), which would put this model at a 0.756 on the Y Axis... Which is apparently between GPT 6.2 Astra and their internal model. Perplexity can also give the same number for a model which understands text as to one that memorizes text... And as I'm pretty sure Inkling-2-Mini is smaller than GPT 6.2 Astra - and it's generally acknowledged that number of parameters equals memorized knowledge, this shows how weak this advertising stunt for OpenAI is.
jrflo 3 hours ago||
The dataset matters a lot here. You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph. If you measured the perplexity on any model on reciting the declaration of independence it would probably be zero.
blueblisters 2 hours ago|||
> You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph

Ideally the hold out set is randomly sampled. You need a hold out set of Inklings pretraining run.

jiehong 3 hours ago|||
Isn’t this a problem? How can we compare and hold companies accountable if they measure this perplexity on different datasets?

Should we trust that the dataset of each company is big and diverse enough to be representative? I don’t hope too much…

programjames 2 hours ago||
Presumably they could each create a graph for their own models and internal datasets, then use public benchmarks to calibrate between them.
ThouYS 3 hours ago|||
depends entirely on the dataset...
poszlem 58 minutes ago||
log10(5,7) = 0,756

ln(5,7) = 1,74

did you even use the right logarithm? because they use the natural logarithm..

edit: that puts it before GPT-3.5 even

bluepeter 3 hours ago||
Let's admit this post has some SEO goals too!
jkoal73 3 hours ago||
in two sentence: „this metric we have been optimizing correlates with the model performance. Big things will happen”
amelius 2 hours ago||
Do we have a clear picture now when the ability to generalize is achieved?
scuppernong 3 hours ago||
this seems to confirm that the model code named "Bel" which solved Navier-Stokes is in fact 6.2 Astra.
hooloovoo_zoo 3 hours ago||
Hmm every annotation but the millennium one could be applied to every point....and possibly that one too lol.
fallingbananna 3 hours ago|
This is not the first time I've seen a chart of "Inteligence on a log scale" related to AI.

And it again feels equally dishonest/bullshitty.

You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.

It's just marketing talk. In reality all three are separate skills and the real intelligence requirements are, if you ask me, not that far away from each other.

To put this in an analogy, it feels like charting "athleticity on a log scale" and putting first running a marathon, then swimming across English channel, then benchpressing 200kg, each one presented as requiring an orther of magnitude being more athletic.

famouswaffles 3 hours ago|
>You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.

Where are you getting 'orders of magnitude'? and yeah I do think it's reasonable to say that Useful Programming requires more intelligence than the median human, and Solving millenium level math problems certainly requires considerably more intelligence than the median Programmer. What exactly is so absurd or unreasonable about that?

discreteevent 2 hours ago|||
> Useful Programming requires more intelligence than the median human

In the same way you could say that beating Gary Kasparov at chess requires more intelligence than the average human and yet we had an AI winter.

clutter55561 2 hours ago|||
The point is that talking, programming and solving math problems don’t lie on a single scale, and each problem requires different “skills.” Thus the comparison with running, swimming and weightlifting.
famouswaffles 2 hours ago||
We have long since known that there is a single factor that explains ~40% of the variation of cognitive tasks and abilities in humans (similar results can be found in other animals). We called that factor the general intelligence factor, or g.

Essentially, Humans who tend to have higher verbal skills also tend to have higher spatial skills, better memories, and faster processing speed. Someone with the cognitive ability to become an exceptional chemist might just as easily have become an exceptional mathematician or software engineer (all else being equal).

The knowledge and skills required are obviously different, but the underlying cognitive abilities that make someone successful in one intellectually demanding field often transfer to others.

More comments...