[1]: https://www.interconnects.ai/p/elicitation-theory-of-post-tr...
Am I right in understanding this to mean that the code on which we're computing perplexity was all written pre-3.5, and so it almost certainly entirely human-written? In other words, this isn't a snapshot of today's internal code base which contains a great deal of AI-authored code.
I would expect the perplexity on the original pre train to increase as reinforcement learning on verifiable domains teaches the model lessons that did not exist in the pre train. In addition alignment and other concerns, unless the are consistently enforced in pre train data quality will also move perplexity up.
The best compression you're going to achieve is when you best understand(i.e. encode) the truth of things.
Perhaps you're thinking of that too narrowly - it is not that the model gets good at understanding how a bridge stays upright; it has to also understand the truth of how a specific person might understand it, and how that specific person might put it into words.
Do that right, as slight deviation of some understanding of how humans respond in general instead of encoding for a single person, etc, etc, etc - and you reach optimal compression.
It's more efficient for the model to encode _how_ an idea is wrong, than it is to encode the wrong idea.
Its how/why they detect when they're being evaluated.
The meta game in pretraining is it correctly assumes the context/mind state of the writer of whatever material under test. To do this effectively requires further modeling and understanding, which is a strong predictor of capability and is a much more difficult task than simply being bad at it.
For example consider this: would it be possible to build a model with such a low perplexity score but it be pretty bad at cyber security tasks? What if we removed all cyber security related data and RLVR from its training?
The article presents an observation but treats it as causation
But, RL improvements are certainly a big part of recent improvements for certain tasks. That's what got the models doing long-running coding and math and research tasks, without constant human prodding.
Why would OpenAI expect any meaningful relationship between their models’ ability to predict the next token of supposedly held-out chunks of their internal codebase and the overall capabilities of the models? I expect both to improve, but that’s about the extent of it.