I am sorry OpenAI, but I think we are seeing OpenAI fall from grace here
Antrophic basically got out before AI bubble burst and Dems block most of the data center build out
OpenAI has been trying to limit its models from quoting whole sale copyrighted works as was the case a while ago, but here this seems like an admission that the path forwards is, in fact, the opposite: to go more towards it. Since obviously, a model that is better at predicting the next token is a model that is better at quoting known text, which might as well be a model better at copyright infringment.
Not that there will be any consequences for them, but an interesting takeaway nonetheless.
"It’s important to note that “compressor” in this sense is something that is capable of performing compression by predicting unseen data, not a compressed artifact like a GZIP file - that is, the best compressor trained on a given corpus does not necessarily “remember” the contents of the corpus."
As such getting better at it will require the opposite to memorisation, it requires generation out of sample.
Also, when perplexity is this low, it arguably makes more sense to report perplexity. A perplexity of 2 basically means that the model can guess a set of size 2 that contains the true next token.
Just some guys on a payroll claiming that an arbitrary set of milestones is a curve towards a greater magic goal that means that some valuation is actually true.
Why would you read this
As said, the labels there are just randomly chosen things that made headlines, thrown on there to imply a clear causal relationship.
That's the level of "science" that gets you banned from campus. In an ideal world, anyway.
There is certainly some influence of the Ghz of your processor on how many matches of StarCraft one can win, but.. well. Bogus science.
That's it? That's all this submission has going for it?
Okay.
The post argues that NTP perplexity is a good first approximation to model capability, and could therefore form the basis for an industry-wide commitment to pause model development at a given perplexity level until there's a consensus that AI alignment and safety has caught up enough that it's safe to continue developing even more capable models.
It's definitely not simply an ad.
The whole AI Bubble topic aside, this is just.. ugh. Intellectually repugnant.
There is just nothing here other than a war of religions. Nothing is to be learned here. There is zero curiosity. It's just a screaming match.
I hate this.
inb4 "no you"
Yeah man. Whatever.
You're not under any obligation to understand it, of course, but if you ever get past this emotional reactivity, you'd benefit from making the attempt, IMO.