Astra can read and write weights to improve Qwen

Posted by dbreunig 2 hours ago

Astra can read and write weights to improve Qwen(rikiyat.github.io)

515 points | 147 commentspage 2

rpcope1 1 hour ago|

So wait so this considered "distillation" and how long until we get another public letter from Dario or Sam pants shitting over this?

O3marchnative 1 hour ago|

Perhaps they'll let it slide if we turn over all our under utilized RAM sitting in our PCs.

iyn 59 minutes ago||

This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.

mynti 1 hour ago||

I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising. Making qwen better at math to me was a lot more surprising

lifty 1 hour ago||

Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.

mlsu 1 hour ago||

I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:

- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.

- the best way to improve on some task is to have interpretability into your own neurons (..?)

- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly

- therefore, models naturally put effort into 'self-interpretability'

I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?

jnwatson 1 hour ago|

Without special effort, a model's reasoning layer doesn't have access to its own weights or activations anymore than we have access to what's going on with our own neurons.

mlsu 1 hour ago||

That's what I mean. Maybe the "special effort" is a 'read your own weights' tool. Or maybe it emerges naturally in CoT. The big labs aren't sharing their chain of thought with us and RL post training techniques are not exactly shared publicly. At the same time, post training is basically the thing that is driving most model improvements nowadays.

jay986 58 minutes ago||

this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.

rumblefrog 1 hour ago||

I wonder if the same technique can be applied to model abliteration? It would easily make any open weight models uncensored.

senderista 33 minutes ago|

It is already the same technique behind abliteration: identifying the direction associated with an observed behavior (refusal, for abliteration) and projecting it out.

happyPersonR 1 hour ago||

Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?

davesque 1 hour ago||

Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?

nater5000 1 hour ago||

>Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics?

Maybe, but it's more like Astra has been trained on enough practical linear algebra problems to be good enough at "eyeballing" how adjusting some weights will result in different outcomes (among other things).

To put it another way: a human can learn to do what Astra is doing here with enough time and assuming the weights are small enough to be practically comprehensible. I know I've gone through the practice of whiteboarding the training of simple neural networks on a whiteboard to understand how they work, so it's not an absurd concept in itself.

The scale of this, though, is certainly extreme and definitely not something that I would have thought any publically available model could handle at this point. It's worth noting, though, that there's virtually no way these frontier labs haven't been exploring this exact concept for a while now.

>In other words, not black magic?

It's just math. Extremely complex and impressive math, but we're still just talking about matrix multiplication under the hood, and none of this suggests anything otherwise.

kingstnap 1 hour ago|||

There is obviously a mechanism to it. Astra isn't magic its an LLM. With perfect interrogation into its training data and weights you could reason about how this happened.

However, this doesn't mean its not magical when someone shows a next token predictor hand tune the weights in Qwen to make it generalize on math.

This is legit the stuff of science fiction if you presented it to mechanistic interpretability people a year or so ago.

aroman 1 hour ago||

“Any sufficiently advanced technology is indistinguishable from magic.”

ernubufiei 1 hour ago|

I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails.

Absolutely incredible regardless.

More comments...