Top
Best
New

Posted by acossta 5 hours ago

Perplexity and Intelligence(openai.com)
394 points | 260 commentspage 9
pishpash 3 hours ago|
I don't understand why anyone needs to exponentiate existing, well known concepts like entropy and cross-entropy to invent something called "perplexity". Somebody has Shannon envy syndrome.
tonyhart7 2 hours ago||
what ??? there is no release model ????

I am sorry OpenAI, but I think we are seeing OpenAI fall from grace here

Antrophic basically got out before AI bubble burst and Dems block most of the data center build out

thevilish 59 minutes ago||
[dead]
blueblisters 2 hours ago||
[dead]
oxcartctl 3 hours ago||
[dead]
natsu529 3 hours ago||
[dead]
twotwoone 2 hours ago||
[dead]
Jepacor 1 hour ago||
The most interesting part of this to me is its relationship with the whole topic of IP Theft/Infringment.

OpenAI has been trying to limit its models from quoting whole sale copyrighted works as was the case a while ago, but here this seems like an admission that the path forwards is, in fact, the opposite: to go more towards it. Since obviously, a model that is better at predicting the next token is a model that is better at quoting known text, which might as well be a model better at copyright infringment.

Not that there will be any consequences for them, but an interesting takeaway nonetheless.

artninja1988 1 hour ago||
They speak to this a bit in the footnotes:

"It’s important to note that “compressor” in this sense is something that is capable of performing compression by predicting unseen data, not a compressed artifact like a GZIP file - that is, the best compressor trained on a given corpus does not necessarily “remember” the contents of the corpus."

Ohentis 30 minutes ago||
The best compressor would just remember the contents of its corpus and compress it to size 0. Which is why metrics like mdl make somewhat more sense.
fergal_reid 1 hour ago|||
That's not how these things work at all. Perplexity will be measured on some held out set, ideally some set of data that has never been on the internet.

As such getting better at it will require the opposite to memorisation, it requires generation out of sample.

amelius 1 hour ago||
By now they probably have a two-step process: first ask an LLM to rewrite the copyrighted work into something different but close; then train on that.
GIFtheory 3 hours ago||
The post makes perplexity seem overly mysterious… First, “log perplexity” is just cross entropy. Why not call it that?

Also, when perplexity is this low, it arguably makes more sense to report perplexity. A perplexity of 2 basically means that the model can guess a set of size 2 that contains the true next token.

hypfer 4 hours ago|
This is just an ad. Why is it top #1 of the front page?

Just some guys on a payroll claiming that an arbitrary set of milestones is a curve towards a greater magic goal that means that some valuation is actually true.

Why would you read this

semiquaver 4 hours ago||
https://en.wikipedia.org/wiki/Flying_Machines_Which_Do_Not_F...
oxcartctl 3 hours ago||
[dead]
AlexCoventry 4 hours ago||
The graph is informative, at least.
hypfer 3 hours ago||
I don't see how it is?

As said, the labels there are just randomly chosen things that made headlines, thrown on there to imply a clear causal relationship.

That's the level of "science" that gets you banned from campus. In an ideal world, anyway.

spuz 3 hours ago|||
Maybe a poor analogy but I see this as OpenAI releasing their GHz speeds or IPC metrics if we were to imagine they sold CPUs instead of AI services. It doesn't tell you much about the final performance but it does give an interesting core data point.
hypfer 3 hours ago||
I don't think that's the case, because with instructions per second, there is a clear relationship. Whereas here on one axis they plot GHz and on the other they plot matches of StarCraft won.

There is certainly some influence of the Ghz of your processor on how many matches of StarCraft one can win, but.. well. Bogus science.

spuz 3 hours ago||
My point was that it is a core but not necessarily correlated metric and it sounds like you agree with that.
hypfer 3 hours ago||
Okay so, the graph is "informative" because it plots the perplexity value of the various OpenAI models over time?

That's it? That's all this submission has going for it?

Okay.

AlexCoventry 2 hours ago||
That was all I'd apprehended of the post when I replied earlier.

The post argues that NTP perplexity is a good first approximation to model capability, and could therefore form the basis for an industry-wide commitment to pause model development at a given perplexity level until there's a consensus that AI alignment and safety has caught up enough that it's safe to continue developing even more capable models.

It's definitely not simply an ad.

AlexCoventry 3 hours ago|||
They're meaningful to anyone who understands the standard numerical objective used in next-token-prediction pretraining.
hypfer 3 hours ago||
I am so incredibly glad that this is going to be over soon, because the amount of weak replies like this is just.. insufferable.

The whole AI Bubble topic aside, this is just.. ugh. Intellectually repugnant.

There is just nothing here other than a war of religions. Nothing is to be learned here. There is zero curiosity. It's just a screaming match.

I hate this.

inb4 "no you"

Yeah man. Whatever.

AlexCoventry 2 hours ago||
The graph shows the average uncertainty the models have when predicting the next token in natural-language text. That's a highly specific measurement, and informative anyone who understands it.

You're not under any obligation to understand it, of course, but if you ever get past this emotional reactivity, you'd benefit from making the attempt, IMO.

hypfer 2 hours ago||
https://news.ycombinator.com/item?id=50602654
More comments...