I feel a lot of people have already addressed the broad stretches to which people are interpreting this. Regardless, the blogpost does open interesting directions that could be useful down the line (2-5years): a) tapping into the embedding space in ways that we don't, ie editing model weights to alter task-performance (mostly preventing degradation) b) Exploring the weight space in more intuitive ways c) Understanding how we can ablate on different parts of the model to understand neuron-based interp better

Seeing this as a replacement of training seems naive, yet one can see this feedback loop targeting weights as an viable alternative to let's say doing LoRA on certain models, especially if we don't want to teach new behaviours.

> And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.

This is basically a claim that the bitter lesson doesn't turn out to be true. Exciting and I want to learn more but I am immediately suspicious.

The potential implications in mech interp of this are enormous. What if we can use frontier models to isolate harmful behaviors?

I was talking to someone from UCL and he said that there was a recent paper that proposed that only 10% (I think) of model neurons are non-polysemantic. I really think mech interp might be on the verge of a breakthrough.

We have all these different pieces of knowledge about model internals but it’s like we haven’t connected them yet.

I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:

> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]

Wow, in a way it's like CRISPR for ML models!

This is as close to scary-self-improving AI as I've ever seen.

Imagine a master malicious language model that produces, distributes, and deploys obfuscated neural networks on the internet. That plus frontier LM hacking capabilities -> large scale distributed embedded intelligence network that no human can see nor understand.

Somewhat similar to obfuscated binaries, but there is something to the fact that NN binaries are less deterministic, and harder to "decompile."

Fascinating.

I was busy using Astra and SOL to build a neural network to play Quake II and do real-time learning of levels and mods so I can play Red Rover or Freeze tage with it without training on it previously. So far its insane what I can conjure up in an evening while I'm waiting for dinner to finish. (If anyone asks, yes I'll write up the huge plan document, source etc. for this when its to a spot I'm happy with)

This survey paper from earlier this year has a lot of good related work:

A Survey of Weight Space Learning: Understanding, Representation, and Generation

https://arxiv.org/pdf/2603.10090 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning

Small annotation on the texts structure:

don’t know other principled ways to hill-climb on text artifacts, >> unlike for updating the weights of differentiable neural networks <<. [...] In this post, we share very early results that allow us to argue that the paradigm of reasoning-driven optimization is now coming >> even for differentiable neural networks. <<

-> Seems like a 'non' is missing at either or.

Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):

* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?

* what, if any, is that extra magic sauce?

* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.