Posted by networked 9 hours ago
In the 2023 discussion of "Demoscene accepted as UNESCO cultural heritage in The Netherlands" I posted a transcript from a video of Will Wright discussing the demo scene:
https://news.ycombinator.com/item?id=36599415
Will Wright Discusses the Demoscene:
https://www.youtube.com/watch?v=m7iuFVmTJus
>You can take any piece of content in the game, and imagine an algorithmic solution to it. Or also, you know, a way that the player could customize that object of thing.
>There's this group in Europe called the Demoscene that make these very elaborate demos for a computer that fit into very tiny little memory blocks, you know like 64K of memory, and you run the thing, and in fact it algorithmically generates about 100 megabytes worth of data, you know these rich 3D environment, generated music, generated wave files, generated animation.
>And they're developing techniques to generate, you know, huge amounts of interesting data, with very very simple, elegant, compression algorithms.
>And this is a skill that game developers used to have, back in the 8-bit days. That was the only ways to do a game like Karateka(?), was to find all these little tips and tricks to compress things and generate them algorithmically.
>But since the CD-ROM came out, and very cheap hard drives, storage is cheap, so basically we've lost that skill set, and now we attack all those problems with brute force. I think we've lost something by dropping that skill set.
[...]
https://news.ycombinator.com/item?id=36613058
[...] Here's a simple low-tech pre-LLM example that shows the equivalence of compression and procedural content generation:
Take a huge text file of HN postings, and compress it with gzip or compress or some other robust compression algorithm. The better the algorithm, the more the output will look like random noise. Then slice the compressed file in half, and replace the second half with random numbers. Then uncompress it. You'll find that at the point you sliced it, it keeps on writing out almost plausible text for a while, consisting of highly probably snippets of commonly encountered words and phrases, then goes downhill towards incoherence. It's not as coherent or confident as an LLM, but the point is to show how low the bar is for using compression for procedural content generation.
LLMs are essentially a form of compression of the world's knowledge or whatever they're trained on, not just word frequencies or pixel patterns, but also concepts and ideas. [...]
Some will say that I should 'judge the idea, not the form'.
But if the author didn't find enough strength to write alone a short ~700 words summary about his work, it means he himself isn't that interested or enthusiastic about it. Why should others bother then? Particularly since low-effort like that signals possibility the whole work is superficial and derivative.
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
The article presents a working experiment, explains why naive decoding fails, describes the beam-search fix, and links the code.
Dismissing all that with presumptuous personal speculation and banal boilerplate drive-by anti-AI snark adds absolutely nothing to the conversation -- and that is intrinsically poor form.
You couldn't even find enough strength to criticize anything beyond the form, while your own form is lackluster.
Ironically, an LLM could have written your comment and improved its form without losing anything distinctive.
and simultaneously you write that 'my form is lackluster' and that 'an LLM could have written your comment and improved its form without losing anything distinctive'. We are having ourselves a small contradiction, aren't we.
Be my guest, enjoy chatbot writing and drowning in slop. But don't encroach upon my freedom to protest it.
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
So I'm encroaching but you're only protesting, huh? I also have the freedom to ironically protest the poor form of your inability to criticize ideas, as well as your poorly formulated unsupportable ideas.
Pointing out someone's contradiction is now being 'critical of form'? And someone's contradicting themselves is 'ironic'?
Now I'm not even sure you know the meaning of words you use. EOT from me.
The fact that gzip is relatively fast should be your first clue that something important is missing.
Gzip is great at predicting the next token for one very specific narrative. LLMs can predict next tokens for entire universes of narratives. Searching for the correct next token across this space scales ~quadratically with the input size. Gzip scales linearly. I can gzip a one terabyte file. Imagine feeding that much into an LLM. These are wildly different animals that happen to overlap in a very small way. Equating compression to intelligence looks increasingly silly to me.
If we must compare language models to compression, they are much more like jpeg and mp3 than they are gzip and flac. I can go fuck with a jpeg file pretty severely at the bitstream level and still have something resembling performance on the other side. Gzip cannot remotely approach this.
In part because gzip only has a 32KiB window size, and I think it'd be at least quadratic within that window if you were going for optimal compression.
Show me an LLM that can run at 300 megabytes per second. Even dedicated ASICs with weights burned in will never move this fast.
Quite well. This project[1], by Fabrice Bellard of ffmpeg fame, is quite old in AI years and uses an ancient LLM, but still beats xz by a solid margin.
A challenge as I understand it is reproducibility.
Normal LLM runtimes aren't typically fully reproducible even with same random seeds for distribution sampling, due to floating-point numbers, batching and such.
Though averaging over many runs could alleviate that I suppose.
While it would measure some aspects of intelligence, I'd argue it fails to capture other, more creative aspects.
And energy consumption.