My PhD supervisor had a beautiful example. Take an English message and ‘compress it’ by removing vowels. It’s still intelligible because English has redundancy:
rrrs cn mk mssgs nrdbl
Similarly, an uncompressed message with errors (swapped characters) is also intelligible because of the redundancy of English:
erwurs lan nake wesaagis unfeatable
But now we do both: we compress the message AND add errors. The result should be much harder (if not impossible) to read:
rwrs ln nk wssgs nftbl
> Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning.
Book (creative commons): https://www.inference.org.uk/mackay/itila/book.html
Lectures: https://m.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWo...
This post references specific parts/calculations, but you'd never know it was not news if you didn't know better.
The author of the post uses standard terminology like entropy coding and arithmetic coding, and cited a paper "in 2023, Google DeepMind released a paper arguing that language modeling and compression are two views of the same thing" which discusses it further.
This blog post is great. Well explained, and clearly took a lot of effort.
I don't interpret it as them claiming to have to discovered it independently.
It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: they fail to cite him)
Either the author knew of prior work that argues the same thing and they ignored it, or they didn't know. And if one writes a 1000+ word article premised on this idea, wouldn't one be presumed to know at least in which century the idea originated from?
Arguably these kind of blog posts should be more subject to such criticisms, because the blog posts purport to "teach" the general public about a concept in an authoritative tone (or at least the author seems to pose as knowledgeable in the subject), while for academic papers, everyone who actually reads the paper knows where the ideas came from anyway and it's mainly an issue of attribution (and maybe about fairly distributing the citation count...)
If the blog is a personal collection of "today I've learned" then different standards apply then "my blog is my research publishing platform".
Both are valid blogs. The fact that it ended on HN front page didn't mean it was targeted towards this audience
This should be expected out of everyone. If you don't respect the reader enough to do this, why should we read your posts?
I think papers should be retracted for not citing prior art, even if you weren't aware of it.
[1] https://news.ycombinator.com/item?id=27440093
[2] https://www.computer.org/csdl/proceedings-article/dcc/2000/0...
Nothing about the concept of "intelligence" is nearly as well defined as the other two, nor as well agreed upon.
Intelligence is a broader concept but definitely involves understanding how a system/envoronment works and making predictions about its unfolding, especially actionable ones that allow you to steer that state towards some goal states.
These are not new.
The con is that you don't get to understand and see any of the history of the ideas or even the ballpark when it was discovered, you attribute it to the blog mentally and you don't know what is how new or old and can't reference it properly when talking to others.
Perhaps you could contact the author with your suggestions?
If you followed the data compression scene in the 80s and early 90s, there were plenty of reinventions of LZ-ish and Huffman-ish algorithms (I also coded my own variant...), and people even tried to patent some of them, so at least for the basics I think it is something that many can discover independently; of course in these times, it's more likely they didn't.
As I say most of these I never publish as I'm mainly using writing as a vehicle for thought, but when I do I'm never sure how to flag them. I don't want (imaginary, lets be honest) readers thinking I'm deluded into thinking I've found something new. I want to come up with a tag I can put on them which adds a pithy disclaimer card at the top or something so I feel more comfortable publishing them.
I don't think it's the case here, but worth noting too that LLM-written blog posts adopt this tone seemingly by default.
Never the least bit of surprise, wonder, doubt, or frustration to get in the way of the steady staccato beat of metaphors, conclusions... and three-item lists.
>> a Developer Educator at ngrok with a passion for nerd-sniping developers.
Maybe more the latter than former...
When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.
You can't just assume smaller is better but it often is. And very often it's more information-dense.
A compression program can compress a book written in a language that the author of the program doesn't understand, on a topic he knows little about.
I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.
This is the crux and reminds me of things like mp3 that exploit the nature of human hearing being limited to a frequency range.
But it's frankly a philosophical question what's real or not. No model is going to capture absolutely everything about the thing it models - at that point, it would be the thing. The best we can hope for is that it captures everything we care about.
And no experiment or metric can tell you if you care about the right things. At best it can tell us if we care about a thing given other things we care about. "No cares in, no cares out".
To make it a little more concrete: you could compress a string from back to front. You could build an LLM to help you do that. If you care about file size, that's almost certainly a bad idea, the forward LLM will be better for that purpose. But are there purposes for which the backward LLM might be better? I think that's not so hard to imagine. Often we wonder about "what came before".
I mean, my interpretation is that the question Turing tried to answer is equivalent to "How can we determine whether machines understand humans/human thought?"
This only works when both systems can talk about pretty much arbitrary things, but if you want a more general method for less complex systems, perhaps having one system simulate another system is sufficient. (Which is also another Turing invention)
At this point, I am unaware of a better definition. I know the Chinese Room argument (and I disagree with it), but I'm not aware whether the proponents of that argument have a better definition of understanding other than "well, the Turing Test isn't enough"...
---
PS: Interestingly the issue of compression is highly relevant regarding the Chinese Room argument -- the essential element in the Chinese Room argument is that the information is not compressed...
No. Teaching is compression. Understanding and application is decompression.
You don't memorize the specific details, you memorize the flow of systems. But you get back to the details to put things into practice.
So was I.
> One decompress it when they can generalize and apply it.
Generalizing is compression. Generalizing and applying are very different things.
> So understanding is compressed, but is not the act of compressing.
Understanding (noun) is compressed. Understanding (verb) is the act of creating understanding (noun) and is compression.
> The larger data it represents never existed.
The larger data is everything you take in about the subject. Experience and instruction.
What is it compressing from?
The author acting as if they discovered this independently had me feel the exact same way. Kinda irritating and almost ... disrespectful? Not sure of the right words to describe it tbh
> The ts_zip utility can compress (and hopefully decompress) text files using a Large Language Model. The compression ratio is much higher than with other compression tools.
It's not only an old idea it's been totally done already.
However, in deep neural networks, you are dealing with compression on different levels of abstraction. How do you decide what shared features a peacock and a palm tree have? At what scale should they be represented? How do you deal with invariance under affine transforms? Do you want to open the box of invariance under non-affine transforms?
When you start looking at what it would mean to compress feature representations, you immediately get to the question of data. You realize that Shannon simply was given a form of a very low abstraction data and that information theory came out to handle data at this level, but it's not suited for the data representations of many higher level modalities.
If you read Society of Mind by Marvin Minsky, which has aged well to about 80%, you can get the hint of the kind of abstractions that humans make and what would be needed to represent them, this is not representable in bits, you need to go to higher level shared features, and then you open all of the questions above as well as credit assignment, mutual information approximation, Fischer information between bayesians, etc.
The whole point is that the representation is learned. When you talk about various levels of abstraction, you're missing that all of these levels are representable with words and the relationships between them. That is verbatim what LLMs are optimized for. Interestingly, when you take an embedding, you do see that some transformations in embedding space actually hold which is quite interesting (e.g. tree + many ~ forest)
2) Language is already a higher-order lossy compressed abstraction made by humans to communicate fast and fill out the left out information with a learned prior. If you train a model on language only, it will not have the opportunity to have a non-compressed representation to make its own abstraction from.
3) If you are LLM-pilled and believe that we will be able to reach arbitrary levels of precise informational representation using language only, and that all abstractions that we may ever want can live on every single embedding layer in an LLM, your argument is fair.
is this not true also of humans?
2) The idea that there is a non-compressed prior is already philosophical no? It seems to just be a rehash of whether an example of a "Chair" is a representation of a perfect Idea of a chair, or whether accidental things put together read as a chair. Its just dualism or not, Plato vs aristotle but for LLMs vs human learning.
If we do happen to just form an abstraction from less examples than an LLM, then its a question of efficiency in the learning algo and the capability to fill the gaps faster/better.
Obvs a world model and multi modal learning would close the gap between LLM latent space and what the human mind encodings might look like. But you can formulate a reality where humans do not have access to a non compressed representation (or even say that doesnt exist) and we are just more efficient at abstraction and recollection
3) There are multiple answers to this. One is that language can just be a prong in the learning modes available. Another is that language is many times redundant and includes supplemental information and nuances that despite being a higher order abstraction has enough additional information to reconstruct entire messages even after the lossy transmission.
Btw I am pretty against LLMs as path forward, but I have been interested in their ability to solve problems with Convergent solutions and their learning model as a lossy compression algorithm where instead of saving the actual data, you save a very good guesser of the next word. So if a text is part of its traning algo then you get almost a 1:1 (like a zip dictionary) but if its not, then he gives you something that plausibly could be in an imaginary Zip with that title.
He wrote https://www.withouthotair.org/ (creative commons) and was the Chief Scientific Advisor to the UK Department of Energy and Climate Change.
Dedicated to "to those who will not have the benefit of two billion years' accumulated energy reserves".
That said, I think it's worth mentioning that Climate Change Without the Hot Air has aged pretty badly, and I'd be reluctant to recommend it to people who don't already have the background to understand what's aged well and what hasn't.
The high level approach of making high level numerical estimates makes sense, but it dismisses solar energy in about a page due to assumed high costs. It turns out that even if you're David Mackay you can still be caught out by exponentials :) I notice now that the version you link has some inline updates pointing out how off the assumptions in this section were, but it seems to me that's not enough; you probably need to redo the entire analysis based on what we know today rather than trying to make purely local adjustments.
On the other hand the point at biofuels are even more inefficient, and therefore a dead end even before you consider broader environmental impacts, are well made and something that is sadly not yet widely reflected in policy.
"adjust the sizes of the available next characters": you mean assigning less bits to characters the most likely to come next, and more bits to the least likely characters to come next?
So it's not totally unlike adaptive Huffman coding but instead of updating the tree on already visited characters, here the tree with the weights of the possible following character is predictive?
Try it out: `apt install dasher`. https://packages.debian.org/sid/dasher
Any rigorous CS program should cover this in depth.
That said, once we do get a working idea of how it works, and can perhaps synthesize a brain artificially with proteins, it will inform us on the next steps for silicon realization of that.
Given the way that models work in 'inference' mode (vs 'training' mode) you can't forward bias the result into the correct result when there are multiple forward results that have identical weights. It's the root cause of hallucinations, and you've lost information in the training phase that you can't then use to discriminate between the 'right' answer and an equally valid 'wrong' answer.
[1] FWIW I could never recover enough state to insure that the image it regenerated was all of the same image you took. So you might get the street but one of the houses might be a house that was in a different picture you took. That kind of bug. Mostly arising out of the same kind of problem you have with using hashes to find documents, when you get a hash collision two documents have the same hash, so you don't know which one to return.
> LLMs do not 'infer' token streams that haven't been trained in their training process
While we're at it, this is simply untrue (in-context learning) unless you generalize "token streams" so radically that it could be readily analogized to humans as well.
[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
The extraction of features from a corpus, the features significant to certain solution, is always and since day zero - compression. As this is the definition of compression - efficient and potentially lossless feature extraction.
So they're both sourcing a bit broader zeitgeist.
This is actively user-hostile. The site is going out of its way to interfere with the most basic possible function of HTML, i.e., the presentation of minimally marked-up plain text. The needless complexity is especially ironic in the context of an article about compression.
If we consider the first example with the arithmetic code, the initial presupposition that only the characters A, B, and C appear in the string already reduces the entropy from 56 ascii bits to 14 bits (A vs Not A and B vs Not B for each character). If you further consider that you only need to distinguish B vs Not B if it's not A, then you can just represent As with a single zero bit and only represent the non-As as two bits (the first of which will necessarily always be a 1 bit). This gets you to 10 bits without even having the proportions of the string. Of course this would be a poor convention if there were say only a single A; in that worst case scenario you would need 13 bits, but simply knowing which character appears the most, without knowing by how much, 11 bits is the worst case scenario for a length 7 string with 3 potential characters. The last bit can be made implicit if you further choose the second conditional appropriately - i.e. if instead of B vs Not B we chose C vs Not C, our last bit would be zero and could simply be dropped meaning both 10 and a single 1 bit encode C - allowing you to encode the example string in just 9 bits and an arbitrary string of that length in 10, again regardless of proportions. That improvement over the arithmetic encoding result in the example is just a case of us cramming a little extra information into the encoding algorithm.
Arithmetic encoding is more clean and more easily extensible, it makes more sense to use than this custom encoding of 7 trits to binary but the point is the "probability" the article mentions is a superficial quality of life feature, not the secret sauce that is the actual key to compression.
It's obvious that an accurate predictor enables encoding only the data that the predictor gets wrong.
But a compressor can encode patterns that defy prediction by looking at the data as a whole. It doesn't have to look at everything in sequence as it arrives.
Applying transformations prior to entropy encoding often isn't just 'rearranging into an easier to compresss format' the transformation can be doing the job of peeking into the future. That makes the encoding a whole lot easier, but it is much harder to call it prediction.
And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).
There is the Kolmogorov Complexity [1], Normalized Information Distance [2] and Normalized compression distance [3] that correlates those.
Finally, there's the Pre-Big Bang Informational Compression and the Delayed Release of Antimatter [4]
All big {rabbit/black} holes to lose some time, if you have any.
[0] https://en.wikipedia.org/wiki/Prediction_by_partial_matching
[1] https://en.wikipedia.org/wiki/Kolmogorov_complexity
[2] https://homepages.cwi.nl/~paulv/papers/chapter08.pdf
[3] https://en.wikipedia.org/wiki/Normalized_compression_distanc...
Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support! Eg: you observe a rare edge case in your training data and (lossy) compression could simply ignore it. But if you wanted generalization in that particular part of the space -- either because an adversary was testing you, or for design freedom where you choose to build in that specific corner -- then you don't just want data compression, but good prediction performance on a test distribution which peaks in that corner.
Assuming that the training data distribution is exactly the distribution you will ever care for is implicitly doing a lot of the heavy lifting in the claim that compression = prediction, and I'm peeved at how much this statement is unthinkingly repeated like a manifesto.
There is nothing natural about the training data distribution, especially if the data generation process is exploratory while the downstream usage will be exploitative.
For instance, consider the distribution of strings drawn from the language '0+'. Now consider the same for the language '[01]+'. A compressor looking at only the strings of the first language within those of the second can do a much better job if it does not have to account for future data.
This also relates distantly to the idea of overfitting in machine learning.
But the Hutter prize (of which I'm a big fan) is for ever-more-optimal compressors, and in fact many of the solutions don't generalise to other input data without stripping out various tricks.
Most of the neural compressors are trying to squeeze every last bit out by sacrificing runtime. You can speed them up a fair bit and they're pretty good. fast-cmix with a small priming corpus will probably be about twice as compact as something like lzma or brotli.
The crucial idea is that once you do that and start optimizing for compression, at some point more compression on the training dataset can lead to worse predictions on the futures you care for. Which is very incompatible with the poster statement.
In that case all bets are off for all machine learning algorithms anyway. If the changes are not arbitrarily different then some of the lost performance can be recovered.
You talk about generalization a lot, but perhaps you are not aware that the technical term is defined under the assumption that test and train distribution are the same.
This means that training data isn't that helpful to top of the line compressors. eg. Fabrice Bellards nncp that's currently the record holding compressor for enwik9 is an LLM that learns on the fly. It's 628KB as shipped (LLM code complexity is high) and it wins on data sets gigabytes in size due to it's ability to create the training data on the fly. No pre-trained LLM comes close when you measure it as above (program size + compressed result).
So it's really not worth that much concern. There's a reason we all say AI is compression and we dismiss comments like the above. They don't pan out. No one's shipping significant dictionaries or pre-trained data. You want to win competitive compression? You'd better have raw code that learns on the fly and is naturally generalizable.
Predictors don't have to give you some corpus-average probability. They can give probabilities adjusted per symbol. The probability could even be exactly 1 for some symbol when the predictor is absolutely certain (and then it costs 0 bits to encode).
If you had a magic oracle predictor that already knew all the data you wanted to send, you'd have perfect compression.
At some level, nothing is representative of all future "problems".
I think as far as corner cases is concerned, people can optimize for what they think matters. Larger models, higher sensitivities, better filtering.
Beyond this, I don't know what you're peeved by. There's no manifesto, there's no claiming that any particular method to prediction is perfect yet.
This is a great experiment. Do some world building offline about Smergle the Keen finding the Hoopla of Kynashiusous. Then ask ChatGPT what reagents would have been more effective in breaking through the borantiastic door than squeeble stomach juices.
Of course we know the answer - it will either drop into a 'I have only been trained on data before blah blah blah' or 'I cannot talk about current world leaders' or 'I have not been trained on your hackneyed and unpublished romantasy saga, why don't you tell me about it?'
On the other hand, if it is aware of the general concepts of your concept, and has bridging concepts for related topics it has been trained upon, it might try to apply the general concepts to a specific but different scenario.
Also sparked the thought that the assumption only holds if the future looks like the present.
This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit?
Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.
Joking aside, the reality is that a pre-Einstein in 1850 couldn't have had the idea of relativity because Maxwell wasn't in his training data.
And a pre-Maxwell couldn't have had the idea of electromagnetism in 1820 because Faraday wasn't in his training data.
What we call "new ideas" are actually just deduced from the available training data, which includes all the previous ideas and all experimental results. AI can do the exact same thing (I assert). Perhaps not today's AI, but some AI, probably not too far in the future.
Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time.
But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data?
The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well.
A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them.
(A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data).
The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
I just wanted to confirm your underlying point here: training a model isn’t about finding a function that fits the observed data (even though that’s the outcome) but instead finding an approximation of the unknown source that generated the source data in the first place.
In the case of LLMs an approximation not of a human but of the sum of humanity that produced the training copora?
And yeah, that means the base model training process is solving the inverse problem of finding an approximation for the processes that generate all human records and also anything else in the dataset with a compressible pattern to discover (weather data, etc). So it's not surprisng that some kind of world model emerges out of this.
I don't think Apple invented the ipod anymore than I invented it; LLMs likely would have also come to the same conclusion about an ipod like device.
Original ideas either dont exist or have a functionally irrelevent definition in comparison with inputing tokens to LLMs to get novel ideas out.
More generally prediction allows you to path find towards a solution to reach some goal that no human might ever see.
If someone states "LLMs can't have new ideas because they only predict" you have to laugh. Prediction literally enables new ideas as you use those predictions to path find to a goal.
The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.
I don't see how that's an exceptional case. Mathematical theories may not correspond to physical objects, but they still have to be tested for consistency, logical validity etc. As for relativity, one could argue that Einstein's theories were indeed not "knowledge" until their predictions were confirmed (like, say, the gravitational lensing thing). The Michelson–Morley experiment preceded Einstein, but it had to be checked that the theory was consistent with that, and so on and so forth.
https://github.com/yuechen-li-dev/GenerativeCompressionProto...
Essentially, copypaste the codeblock in the Markdown into any LLM chat, and it will return with the benchmark results. Very easy benchmark to run.
Essentially, semantic compression refers to reducing the size of a set of data while retaining its full semantic meaning. The useful application of that is of course, with prompt compression to save context. I know a lot of people essentially sends their prompt to another LLM to compress into JSON first before they send it out, and this came out of an experiment to see the best method to accomplish that task, and the idea is that the compressed and uncompressed prompts will return the same result if sent to another LLM.
What that block of Chinese text is essentially a kind of "meta-prompt" that causes the LLM to reflect on itself as well as the method of how to compress information into the highest possible density form, and the reason it is in Chinese is because it is the language with the highest semantic density that I know of. You can ask an LLM to explain what the text in the block means to have an explanation of what everything means and why it works, but overall it tends to greatly increase the efficiency of semantic compression task of turning prose to JSON across the board on pretty much every LLM that I've tested it on.
That's basically the explanation of it, I thought it was a crazy discovery when I found it a couple of months ago, but now I just think it is pretty neat.