I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.

They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.

The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?

"No human can go edit a brain to become better at math" -- I'd offer this is exactly what "learning math" is.

I know I'm being pedantic, and I understand you mean "edit" like "surgically", but it does still feel worth noting.

My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.

EDIT: I think the bigger implication is that backprop is not the only game in town.

It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.

I am very shocked by this as well, although it also kinda feels like it makes sense, you know what I mean?

If this turns out to be as big as it sounds, in hindsight it will look very obvious, I feel like!

Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.

So I just asked Astra if it thinks it could do this and it said no, it can't.

Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?

> No human can go edit a brain to become better at math.

For now...

I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.

Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.

I have so many questions.

Could a model then transmit concepts to another?

Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?

What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?

Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.

Why is it so surprising? Humans can make compilers that are better coders than they themselves are.

All the labs are training these models on LLMs/machine learning like crazy.

Remember that they want to build ASI, these LLMs are just intermediate steps to help them get there.

its a program optimising a program it has nothing to do with a human reprogramming their own brain

I'm baffled this is surprising. This has been the goal and every data point on the trajectory has been leading here and beyond. Opus 5.5 was created by many research engineers using Opus 4.6 extensively. Opus 6 is being created by using even more intelligence and agents on Opus 5.5. They are approaching autonomy and Opus X+1 will soon be rapidly created autonomously from Opus X. Given we have practically zero interpretability into the models, and we have not solved alignment, they will recursively improve out of our control. The engineers in A.I. labs are actually generally scared of what's unfolding. They don't know at what point this RSI takeoff occurs and what will happen after.

I mean, pattern recognition and manipulation is what these models are ideal for.

It is a very cool discovery, but I'm not surprised that you can do it.

You acknowledge that you are working on something similar, yet refuse to elaborate and vaguepost...but there is enough information just from your post alone for a dedicated actor to attack you and extract such world-ending information ("world-ending" implied from your ominous reply). Would this not make you as naive as the team that published their findings?

I applaud the authors for sharing their findings and showing real results. Rather than write some bait I would find in an influencer-peddled tweet and allow rampant speculation. It would have been better for you to say nothing.

The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.

I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.

You don't think an LLM can perform tasks not in its training data?

>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?

I mean, this question indicates that your understanding of how these models work is, indeed, lacking.

Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.

I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.

damn i'd imagine so too! didn't think of that. I'd have to imagine there's an increasing challenge with the number of weights involved but who knows how it shakes out

You can bet that the world is about to spend a lot of tokens trying to answer that question.