This reads as written by someone who just happened to understand what LLMs do, so I totally fail to understand how anything further said can have any real credibility…
As a matter of fact the best compression by Fabrice Bellard’s models have been achieved with NNs long before LLMs.
And also MP3 and MPEG in general are very apparent neural networks, yet not deep as in modern VLMs
https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
Also interesting the top entry is from fabrice bellard: https://bellard.org/nncp/nncp.pdf
Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
You can, actually! Any compressor can be losslessly converted into a generator, and vice versa.
Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.
Makes me wonder idly, - is this conceptually akin in some sense to a "universal grammar," and if so - with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training, - does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics
But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc.
This paper proposes something similar where you can inject memory without training https://arxiv.org/abs/2605.16893
However, going from compression to prediction is a large jump that is unsubstantiated by this article and based on the claim that probabilistic recall is also prediction.
Two perfect counterpoints to this are markets and weather patterns. One cannot predict future events based on past performance or behavior. Change is the only thing that's constant, and chaos/entropy is everywhere we look.
For simple problems like programming, sure predictive recall works amazingly well, but let's not pretend LLMs are actually predicting something. This is exactly why LLMs suck at doing anything novel; they lack imagination and creativity.
And a counterpoint to your counterpoint: If it was raining 10 seconds ago, it is very likely that it is still raining right now. There... you can make a prediction about weather based on historical data. It's like perlin noise, random but with discernible patterns
I think this is relevant to the discourse on LLMs/programming because for months, people said “they’re just regurgitating their training set,” but now I think people are seeing (I am seeing) that they do learn more abstract models of the world than that. I don’t really know how, but it’s why they can generalize from other codebases and tools and so on.
Stock market is even more pathological than weather here. Most statistical tools have baked-in assumptions about some degree of independence between factors under analysis. In weather, things are hard, because everything affects everything but in some random sense. With stock market, it's worse, because some things affect others in an adversarial way. A pattern discovered is not merely drowned in noise, but rather actively canceled out as other people discover and act on it. The way to describe this in formal terms is, I believe, that the stock market is "anti-inductive".
Fortunately, most things we deal with life are much friendlier to predictions than weather or stock prices.
> I think people are seeing
Better late than never, given that this has been empirically demonstrated two years ago by Anthropic. I actually don't understand why people don't make a bigger deal out of it - maybe it's because the AI hype/hate fight is sucking all attention from the room. But the point is: we not only have a thing that can learn concepts we recognize and build abstract world model, but also we have a degree of visibility and control of it we never had before. We can't see, much less suppress or stimulate individual concepts or concept groups in a human brain directly, but we can with LLMs.
> A prediction is a statement about what you think will happen in the future, often based on experience or knowledge. It can also be referred to as a forecast or an informed guess
Based on my reading of this definition, compression may inform prediction but it is not itself prediction. The examples cited in the blog post are examples of probabilistic recall based on past events or instances. More context means a higher chance that the recall is more likely to be aligned.
But it's hard for me to accept the leap to compression == prediction because in my mind a prediction is an informed guess about something that hasn't yet come to pass. But thinking more about it, time is a human concept and so who's to say the temporal reference means anything at all here. Maybe probabilistic recall is the same as predictive forecasting if time is an invented concept and essentially means nothing?
Is everything fundamentally deterministic if you know everything in the universe or does free will exist?
IDK to be honest, I'm just more frequently surprised by new things that happen every day than I am at things that stay the same, even if mostly things stay the same. Maybe I just don't notice them and nothing actually ever happens.
Side note: the inevitable consequence of this line of reasoning will eventually become that LLMs given enough power are in fact intelligent and sentient, and I'm worried about how that affects humanity as a whole. Are we about to subjugate the most intelligent thing humanity has ever created, or is it about to subjugate us?
The rabbit hole gets deep quick when making the leap between a fancy recall mechanism and novel prediction, but I agree they're not that different in the end. I just believe it's important to be nuanced or else we'll miss when AGI actually happens (maybe it's already here).
The article itself has decision trees for the compression explanation, which is also a lookup index.
In each case you try to recognise (re)usable structure.
Self-indexing succinct data-structures are a good example of the third side of the coin.
So it's a trinity: compression, prediction, indexing
Why only LLMs? All statistical models are compressor. You can say "model" and "compressor" are synonyms.
Article does not mention "embeddings" at all, even though it's commonly viewed as a compression method. Also "encoder" part on "auto-encoders".
Predictability is the inverse of information density.
Low information density enables high compression, and vice versa.
It's called entropy. This is basic information theory to be quite frank..