Posted by dbreunig 1 day ago
Consider an analog - Human brains are made of neurons, and so are fruit fly brains. We now have a fully connectome of drosophila melanogaster, so shouldn't we – very smart brains – be able to look at the fly connectome and make sense of it by just looking at it? Of course not! That's friggin' crazy. Our neural weights have been trained to take in data from senses and to produce output behaviors (moving muscles, squirting hormones).
An LLM has been trained to take in text and images and generate "good" output text. It doesn't natively understand what the 17th weight on the input channel implies. Instead, when it sees "strawberry" what it really sees are 3 symbols "st", "raw", "berry", and these symbols are really just identifiers [302, 1618, 19772] which get converted to basically meaningless numerical vectors on the input plane. This is why in 2025 the frontier models kept swearing to us that "strawberry" had 2 r's!
So when the model sees the full representation of a pacman playing neural network (here's the actual prompt they used https://lnkd.in/e5Td2amH) it reads in a bunch of numbers like "0.054 -0.026 0.014" as text. This isn't somehow presented to the LLM it in a "native" neural weight form. Rather, it gets tokenized just like everything else, into meaningless symbols – in this case "0", ".", "054", " -", "0", ".", "026", " 0", ".", "014". Nevertheless, the models are able to read reams of numbers and pick out patterns that let it know that it's looking at a pacman playing neural network.
Getting back to my conspiracy theory. This new talent for LLMs to read and understand model weights is unnatural. You won't find much training data available on the internet that is analogous to this. Humans simply train neural networks. We don't look at specific networks and write essays describing how model weights lead to various model behaviors.
This new talent is so unnatural that I posit it is actually being trained into the frontier models intentionally. The frontier companies are actively experimenting with models that are capable of reading models and directly understanding and modifying weights. OpenAI and Anthropic have known this capability existed prior to the report because they are the ones that put it there!
I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.
I simply must know now: Could they do it with neurons in a brain?
Some absolutely crazy and insane experiments if this ability grows.
Could they edit an animal's brain? Can they teach things to animals???
It works! In mice