There is a correct sense, but we're sort of garbling concepts here:

Predictability is the inverse of information density.

Low information density enables high compression, and vice versa.

It's called entropy. This is basic information theory to be quite frank..

A few years ago I published BIDEN: Binary Inference Dictionaries for Electoral NLP, based on this idea - https://matthodges.com/posts/2023-10-01-BIDEN-binary-inferen...

Unrelated to the content: I was really pleased to see that this site defaults to the bare minimum for cookie consent. I reflexively clicked "Reject all" only to see that it was already the default, which threw me off.

The comparison can be carried on to another even crazier level: Evolution is compression. All the complexity of biology is executed at the highest possible efficiency.

It's more or less obvious that the LLM is a lossy-compressed version of the training data; it reproduces sequences of tokens that are the sort of thing that could plausibly occur in the training data, and avoids sequences that are implausible.

Because most of the training data has good grammar, the LLM is strongly trained on grammar; it will rarely predict ungrammatical gibberish. Even if there are grammar mistakes in the data, they are not systematic and so don't reinforce each other.

1) am I allowed to scan the entire corpus in advance before I populate the dictionary? Is this a stream, or is their an EOF marker I will know in advance?

2) if 1) then "prediction" isn't the word I'm looking at.

The prediction happens when it is uncompressed right?

LLM embeddings are compressed training data.

To decompress that is to make a prediction (in this case to convert the embedding into readable text)

I tried asking a zip file to write a program for me but it did nothing.

I’m starting to think that compression is not, in fact, prediction.