This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?

I'm working on replicating these experiments (and building the tooling in the meantime) starting with Experiment 8 (the Qwen reasoning loop fix).

My repo: https://github.com/openwsp/openwsp

The forward passes are in progress, 5/50 as I write. It's costing real money and time, and I'd love to collaborate with other hackers (find my email first).

Check my understanding: Astra and Qwen would likely choose a different number— weight— to represent any given concept, like “king”.

So to fix the weights, Astra first has learn a new language of how the numbers that represent weights map to meanings?

Showerthoughts:

1. If you know ANN or, say, perceptron, you know you can eyeball models too. You can even have certain strategies in composing your own network. This weight-space programming thing is nothing new in that sense, and the article doesn’t really provide much beyond “LLM can do it”. That’s not even science…

2. I’m pretty sure this is Transformers thing, not even LLM. Transformer is, by itself, good at finding patterns in relatively-larger data. So this is unlikely limited to Astra.

3. This approach quickly becomes impractical as the scale goes up. Sure, you can also have multiple models chained with fewer connections, but, now that’s what we’ve been doing for years.

All in all, yeah, an interesting find, but too far from changing the game. Over-exaggerated every way. Perhaps one should start from formulating what weight-space programming is, what methodologies, practicality, etc. LLM bench is one of the last things you need.

I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails.

Absolutely incredible regardless.

This is remarkable and something I would have never though of doing regardless of how much it makes sense!

I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.

Let's get sci-fi and ponder.. weights programming replaces conventional training. Is something in the veins of the Ken Thompson Hack conceivable? A NN secretly injects its weights that "procreate"?

If we are being slightly reductive, this is just more of the exact same thing. The token stream in this case happens to be a projection of the weights of a neural network. I don't see where we violated any of the prevailing assumptions about how these systems operate.

The mapping between spaces in this case is mostly linear. This isn't like gzip or a Turing machine. I'd argue this is even more permissive (easier) than the LLM's originally intended use case of human language.

What happens if we can come up with a model that can directly emit a frontier LLM given some seed context? I feel like we could train a model to recreate astra verbatim. At some level it's just bytes on a disk that can be bundled up into tokens. As long as the representation is rational and consistent I don't see why this wouldn't work.

I dont feel the air of impressivness here. It seems like of course so. Of course theres geometric relationships in the data we dont understand, of course a mathimatical relationship machine found it.

These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?

What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.

I wonder if the same technique can be applied to model abliteration? It would easily make any open weight models uncensored.