Previous discussions of that specific page https://news.ycombinator.com/item?id=48557691

I'm more interested in the converse question: how well does an LLM perform as a compressor, compared to gzip (ignoring its insanely lower speed)?

R. Hendricks, D. Chugtai, and J. Dunn, "Lossless compression via optimized middle-out bitstream processing," Pied Piper Inc., Palo Alto, CA, Tech. Rep. 42, Apr. 2014.

I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:

  gzipt \
      --corpus data/tinyshakespeare.txt \  
      --prompt $'MENENIUS:\n' \
      --length 200 \
      ;
  
  MENENIUS:
  MtLUMSeptuttyyyxyxyxyxyvyyyxyxyxyxyvyyyxyxyxyxywyvzyxyxyx
  yyxyyyxyxyxyxyxPlyxyxyxyxyxyxyxyxyxtoxzfTUS.zxzzzyzzzvzzz
  vzzzxvzyvyxyxyxyvyxyxyxyvy--,Vdvyxyxyxyxyxyxyxyxyxxy!zFlx
  zzyyxyxyxyvyxyxyxyvyySPffuyuy

Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...). Why are they runs of alternating symbols and not one symbol?

Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."

  @@ -33,19 +34,16 @@ def candidate_lengths(
       level: int = 9,
       pool: ThreadPoolExecutor | None = None,
   ) -> list[int]:
  -    """Compressed length of ``context + seq`` for each seq, sharing the context.
  +    """Compressed length of ``context + seq`` for each seq.
  
  -    Compresses ``context`` once into a ``compressobj``, then clones its encoder
  -    state per candidate and feeds only that candidate. Identical to
  -    ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
  -    match search over ``context`` happens a single time.
  +    Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
  +    snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
  +    whole block, so every candidate recompresses the full context. Threads
  +    still scale because ``bz2`` releases the GIL.
       """
  -    base = zlib.compressobj(level)
  -    head = len(base.compress(context))
  
       def length_for(seq: bytes) -> int:
  -        clone = base.copy()
  -        return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
  +        return len(bz2.compress(context + seq, level))
  
       if pool is not None:
           return list(pool.map(length_for, sequences))

Interesting approach, I wonder how this could be used as a classifier. :)

Good article.

Compression is a property of language.

A seemingly simple sentence like "I had lunch" has enormous amount of information compressed inside it.

The word lunch is a compressed form of "having food at noon" while "noon" in turn is a compressed form of "Sun's position against Earth's rotation" and so on and so forth.

Every sentence has layers of compressed sentences. How many layers one chooses to decompress is up to the person.

Yay, another mostly AI authored piece with vibe-coded aesthetics.

Some will say that I should 'judge the idea, not the form'.

But if the author didn't find enough strength to write alone a short ~700 words summary about his work, it means he himself isn't that interested or enthusiastic about it. Why should others bother then? Particularly since low-effort like that signals possibility the whole work is superficial and derivative.

Only if it can run DOOM.

Will Wright once described the same connection from the opposite direction: compression as procedural content generation.

In the 2023 discussion of "Demoscene accepted as UNESCO cultural heritage in The Netherlands" I posted a transcript from a video of Will Wright discussing the demo scene:

https://news.ycombinator.com/item?id=36599415

Will Wright Discusses the Demoscene:

https://www.youtube.com/watch?v=m7iuFVmTJus

>You can take any piece of content in the game, and imagine an algorithmic solution to it. Or also, you know, a way that the player could customize that object of thing.

>There's this group in Europe called the Demoscene that make these very elaborate demos for a computer that fit into very tiny little memory blocks, you know like 64K of memory, and you run the thing, and in fact it algorithmically generates about 100 megabytes worth of data, you know these rich 3D environment, generated music, generated wave files, generated animation.

>And they're developing techniques to generate, you know, huge amounts of interesting data, with very very simple, elegant, compression algorithms.

>And this is a skill that game developers used to have, back in the 8-bit days. That was the only ways to do a game like Karateka(?), was to find all these little tips and tricks to compress things and generate them algorithmically.

>But since the CD-ROM came out, and very cheap hard drives, storage is cheap, so basically we've lost that skill set, and now we attack all those problems with brute force. I think we've lost something by dropping that skill set.

[...]

https://news.ycombinator.com/item?id=36613058

[...] Here's a simple low-tech pre-LLM example that shows the equivalence of compression and procedural content generation:

Take a huge text file of HN postings, and compress it with gzip or compress or some other robust compression algorithm. The better the algorithm, the more the output will look like random noise. Then slice the compressed file in half, and replace the second half with random numbers. Then uncompress it. You'll find that at the point you sliced it, it keeps on writing out almost plausible text for a while, consisting of highly probably snippets of commonly encountered words and phrases, then goes downhill towards incoherence. It's not as coherent or confident as an LLM, but the point is to show how low the bar is for using compression for procedural content generation.

LLMs are essentially a form of compression of the world's knowledge or whatever they're trained on, not just word frequencies or pixel patterns, but also concepts and ideas. [...]