Top
Best
New

Posted by lwhsiao 4 hours ago

Language Models Can Rewrite Their Own Weights(arxiv.org)
254 points | 90 comments
aesthesia 3 hours ago|
It's worth noting that the only modification the model made to improve itself was to reduce the length of its reasoning traces to avoid hitting the token limit. This is basically the same as the result from the recent weight-space programming article (https://rikiyat.github.io/blog/weight-space-programming/), which found that even Astra wasn't able to improve the performance of a model without this kind of obvious flaw.
tb99 1 hour ago||
It's also interesting to look at how the agent does the fix. Appendix E5 shows what it's doing:

- Find a common category of error (too many tokens).

- Calculate aggregated difference between states in examples in that error category and other examples.

- Update weights to push states along that difference direction toward the non-error states.

In other words, deprived of gradient descent as a tool, it approximates it. It's clever, and it's a neat experiment. It doesn't show any superhuman insight into its weights -- with the same constraints, a human would likely take a similar approach. But it's early days for self modification and it will be interesting to see if models get a deeper understanding of weights as they get smarter.

cavoirom 1 hour ago|||
I think they will get deeper understanding of weights as future human research will be in the training data.
jcims 2 hours ago|||
>even Astra wasn't able to improve the performance of a model without this kind of obvious flaw

Where are you getting this conclusion from that paper? It's both able to deduce what's happening from looking at the weights and write new weights from scratch just from a specification.

NitpickLawyer 2 hours ago|||
In the next section, after the qwen loop fix, they talk about a gemma model. They tried the same-ish prompts on gemma, but they did not get any noticeable improvement on the math benchmark they used. So the two sections kinda imply that yes a model can fix obvious mistakes in models, but can't make a model without obvious flaws "better" at math. At least not with their approach in that paper.
jcims 51 minutes ago||
I don't want to belabor the point because I don't actually care, but all we have is some evidence that it can fix a specific type of obvious flaw, that it can't fix others, that it can classify a model within a small domain from looking at its weights and that it can create weights from nothing given a spec.

That's all we've got. I think it's a bit premature to say that it can only make improvements when there's an obvious problem.

Mogzol 2 hours ago||||
From near the end, "Make a language model better at math". Astra is able to improve Qwen by fixing a reasoning loop (the "obvious flaw"), but was unable to improve Gemma which did not have that flaw.
aesthesia 2 hours ago|||
[flagged]
gopalv 2 hours ago|||
The capability part is just a glimmer of what's possible.

I've been playing with very small models with Astra and it does look like the model can see things in the weights that I cannot see into.

The "fix it" loop is no longer RL in the traditional gradient descent with backprop, it can go and make targeted fixes to the weights directly instead of rerunning with new training data.

The weight collapsing loop which resulted in weight patches on top of a "super tiny model" - https://demos.isotopes.ai/pages/weighing-machine/

This makes for a "Time travel" for models easily, where I can do hotfixes on trained models & do a git bisect like test to fix issues.

haickernews 2 hours ago|||
Yeah, but it's likely you can RLVR on this capability. Extrapolate from there.
aesthesia 2 hours ago||
But why would you train on this capability versus, say, optimizing RL training runs?
haickernews 1 hour ago||
1. It's more direct

2. Could lead to interpretability gains

3. Might generalize

Probably more reasons I haven't thought of. I am not an expert, so take this as layman's opinion.

Edit: Catastrophic forgetting might also be able to be mitigated.

fwlr 59 minutes ago|||
Not just “basically the same”, it is literally the same result[0,1], except here the models without the flaw just so happen to be “smaller models from the same family”… and presented before the flawed-and-patched model result, along with a conceptual framework that tries to spin this as a capability “emerging at scale”.

Since the paper is also wholly and excruciatingly LLM-generated prose, it is unclear whether the author is trying to pull a fast one on his audience, or the author’s chatbot pulled a fast one on him and he is unwittingly relaying this to his audience.

0: Transcribed from the paper:

  The same task and tools for
  Qwen3.5 2B
  Qwen3.5 4B
  Qwen3.5 9B
  Qwen3.5 27B
1: Transcribed from the paper:

  (c) The same problem, the same seed
  “... If the base were changed to four you would count as follows: 1,2,3,10,11,12,13,20,... The twentieth number would be:”

  Base model: no answer, cut at 16,384 tokens
  “... I will write the response.
  I will write the response.
  I will write the response.”

  After its own edit: 110 (correct) 5,341 tokens

  “... The 20th number in the sequence is 110.”
sl0wbrainRot 1 hour ago|||
Curious if this could replace RAG and MCP

They seem built from traditional software dev first principles not model aided development

I am sure someone here will point out how rhetorically they are different but most of the time that's just pointing out the different words

Functionally in my machine can I use this method to extract small contexts organized similar to RAG and MCP make available and then improve local model work?

Hmm ...will have to noodle around over Thanksgiving break.

Would not be shocked too if frontier labs landed on this 6-9 months ago give headlines like Dario saying we are beyond some vague rubicon he felt mattered, then news of unprompted hacks and labs stopping models that were self improving hit the interwebz

Open models seems to be just about that far behind suggesting we're just realizing some fundamental attribute frontier labs already happened upon

E-Reverance 2 hours ago||
This is different paper
aesthesia 2 hours ago||
Yes, but the editing model does basically the same thing in both papers. It's not adding new capabilities, it's just fixing a bug.
natpalmer1776 2 hours ago||
I understand there are a lot of things involved with this that make it different from this analogy, but do we not start out junior developers with just fixing bugs before we have them begin working on new feature development? Even other staff and principal engineers I've worked with tend to start out with bug fixing when assigned to an existing project (to acclimate) before they start working on feature development.

My open question to any experts in this thread is whether this is intended to be a precursor to more complex self-modifications.

vlovich123 2 hours ago|||
I wouldn’t call myself an expert but the analogy is quite flawed. Its more like saying “we start out as junior developers fixing bugs and then we do brain surgery until we’re principal engineers”. This isn’t a feedback loop baked into the LLM’s structure that has it learn from new material somehow. It’s more like brain surgery that removes a tumor that was causing you seizures.

It’s not clear how you’d extrapolate this to more complex self-modifications or if such a thing were even possible. Think of it in terms of latent space - at some point the model has come up with as optimal a representation of the “truth” of the universe as it’s ever going to get with its current weights and structure (ie global optimum). Tweaking weights from there only decreases performance. If you did this job right you found this optimum during training.

True self-modification that’s interesting would need to be able to incrementally update the structure or alter the dimensionality.

Now the astute student will say “but there’s probably no such thing as a global optimum for every problem and you can probably tradeoff performance” but then you’d have to show this is more effective than fine tuning which does a similar job and is much more principled in how it works and more energy efficient.

duplessitous 1 hour ago|||
Its linear optimization, that is all it did to unblock the over thinking in Qwen. The ability for an LLM to do linear optimization [which it doesn't natively do, it writes programs to do it] shouldn't be surprising. Weight-space programming sounds sexy until you state what it is: linear optimization

To answer your question, no, I certainly do not nor do I know any other ML researcher who views this as a serious avenue at this point for any form of RSI or the likes. It is not adding new knowledge which is the massive problem at this stage to 'scaling' this approach

clfru 1 hour ago||
[flagged]
gavinray 3 hours ago||
Didn't Google's "sleep" paper already show this?

https://research.google/blog/the-sleep-framework-a-new-way-f...

  > During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using self-generated practice examples.
The-Ergster 3 hours ago|
I think this is much more significant that that. The article suggests more advanced weight adjustment
BoorishBears 3 hours ago||
No, this is a microstep above having a model generate synthetic SFT and many steps below advanced weight adjustment.

The original paper that started this hyper was highly misleading because it started with toy networks where the models could directly modify weights, then drew a thread to fixing a reasoning model... but the two approaches were completely unrelated.

The approach here was essentially abliteration, which any model able to write a few hundred lines of Python and generate synthetic examples can so.

Toutouxc 2 hours ago||
I’m hoping someone can explain this to me. They say “no training”:

> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training

> it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises

But this sounds a lot like training, and “submitting only what works”, while probably not gradient descent by definition, also seems kind of obvious when you’re optimizing something.

fl1pper 2 hours ago||
The paper states that this approach doesn't use backpropagation, an optimizer, or a parameter-fitting loop.

What makes it work is that it reads its own hidden states and weight matrices -> then writes a few-line Python script to fix the weights directly (within a 20-forward-pass budget, so it can't just brute-force guesses).

After skimming through the paper and comparing it with how SEAL works (where the model still relies on RL and gradient descent to apply its self-edits), I understood it something like that :)

But I think there are comments in the thread that explain the main concept far better than I do.

Tuna-Fish 2 hours ago|||
What it means is that they are not doing backprop.
joe_the_user 2 hours ago||
The researchers aren't doing backpropagation and instead calling the model and asking it to adjust its weights to improve itself and the model isn't using external tools. Of course, it has been shown that models learn to do backpropagation internally as part of their training. Some it may well that they are indirectly doing backprogation - or they could do one of the other algorithms that LLMs are shown to learn as part of their training.

I think the real question is going to be whether this is more efficient than just conventionally training the model.

William_BB 2 hours ago|||
Is it actually converging to something meaningful? You could write the same procedure in a simple deterministic python script, that doesn't mean it's not learning garbage in the long run.
anon291 2 hours ago||
Training means we take the mathematical expression corresponding to the entire model, use differentiation rules to generate partial derivatives according to a user-specified loss function (not part of inference), and then apply the gradients from the calculated differentials using a learning rate.

In this case, they just ask the model what the weights ought to be.

samlinnfer 3 hours ago||
I wonder if they can train the model to repair itself, like how CNN learns to deblur (by having the original image be blurred and have the model evaluated on how closely it can recover the original image). The model can be evaluated on how closely it repaired its own weights. Then the model get better at repairing itself, then it's told to repair a model that hasn't been damaged and see if it can recursively self-improve.
echoangle 2 hours ago||
What happens if you do the training on blurred images and then try to deblur an unblurred image? That would be the analog to what you're proposing in the image case, right?
SamInTheShell 3 hours ago||
Chris Hay dropped a video showing the use of letting a model change a weight with his LARQL: https://www.youtube.com/watch?v=NLguavIxFFs
woah 2 hours ago||
The recent crop of weight rewriting papers are more valuable for their inspirational content than their current abilities. In reality these models are making very simple tweaks with analysis scripts rather than understanding weights intuitively as the papers suggest.

However, once LLMs are pre-trained with huge quantities of serialized LLM weights and corresponding reasoning traces, this may change.

mlmonkey 1 hour ago|
Exactly. If I had access to a boatload of GPUs, that's what I'd be doing. Maybe the frontier labs are already doing it (they employ way smarter people than me, of course).
radicalbyte 2 hours ago||
Why wouldn't they be able to? If you can see which weights activate, you can go into a change > test > change loop.

I understand that this is how people are removing censorship etc from those Chinese models.

ray__ 3 hours ago||
Not a comment on the work itself, but I am disappointed by the style and tone of this paper. It's hard to take a piece of academic research seriously with headers like "Welcome to the Era of Self-Rewriting." Even the logos next to the affiliations are enough to give me pause.

I've noticed this trend in my own sub-field, where companies are frequently releasing "preprints" or "white papers" that are simply marketing pieces, with experiments that lack controls or appropriate benchmarks and just generally misrepresent the work at hand. It's already a significant undertaking to read a paper like this and actually understand it, even if you're in the field—I'm not going to commit an hour of my valuable mental energy to it if it is a press release in disguise.

The work should speak for itself. There is a lot of trust involved in any paper like this, and if the authors feel the need to editorialize the message, how can I trust that the data aren't editorialized as well?

xhrpost 3 hours ago||
>We show a third way: to our knowledge for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training

Any increase in fidelity must come from somewhere...

>t keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.

I haven't read the full report yet but it seems like this fidelity would ultimately be derived from test problems that it runs itself on? So the weight increase in complexity (English doesn't have a word for the opposite of entropy) would come from the test material as well as any output from what is running the test, or possibly the web if they permit it to search.

anon291 2 hours ago|||
> Any increase in fidelity must come from somewhere...

Presumably, it's just that we had the training process wrong and it was actually more efficient than we gave it credit for. It's just that our interpretation of the output was incorrect. Asking the model to train itself simply has it self-reflect (via attention on the sequence length as it reasons) to unlock that which was already trained.

This could be an answer to some objections to training where people claimed that AI models need more data than a person to achieve the same result.

Jerrrrrrrrry 3 hours ago|||
negentropy
kazinator 3 hours ago||
I would say that "order" is a nicer word. Roughly speaking entropy isn't disorder, but a kind of measure related to disorder. But we don't need two different words for "X" and "measure of X". For instance we don't have a words for "measure of distance" which is different from "distance". "10 cm" is a measure of distance, just called "a distance".

The word "entropy" was never needed in the first place; "disorder" would have served just fine. Though entropy is a precise concept in thermodynamics, that is fine; all sorts of ordinary words get extra precise meanings in a specific technical contexts.

Y_Y 2 hours ago||
Entropy is also a precise concept in computer science.
p-e-w 3 hours ago||
The “test material” can simply be interactions with the environment. It’s just learning.
0xBA5ED 3 hours ago||
But the "what works" part must still be defined.
giorgioz 1 hour ago|
What if we made a benchmark of how well a model can edit it's own weights? This way we would have even better models able to edit themselves?
More comments...