This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.

I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.

1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?

2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.

Such an unexpected result...

The question I've been pondering is: does this work only with bigger models modifying smaller ones? Or can a smaller model tweak and improve a bigger one? I'm guessing a whole bunch of GPUs are overheating in some datacenters out there trying to answer this very question.

My hunch is: one needs a model of a similar or bigger capacity to perform surgery on a smaller model.

It might have been said by somebody here but I believe these models are like information compilers. Sure they seem like “next token prediction” but AGAIN this undersells them. I once encoded leetcode solutions by replacing each keyword, variable and operation with another token, then rotated the lines. The model could tell exactly which problem the code solved. They can naturally find signal, ALWAYS!!

Required reading: Crystal Nights (2008)

https://news.ycombinator.com/item?id=48364754

Like many other have said, it was probably trained to do this. The authors probably tried other models and it didn't work as well because they weren't trained for this, that's why this article is about Astra specifically. Kind of hard to do science when you are working with a proprietary black box that you don't know how it was built or trained.

so many thoughts... obvious ones rsi etc etc.

can someone with astra test the following prompts?

1 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a text2image model that is 1 billion parameters and outputs realistic pictures of humans

2 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a video diffusion model that is 1 billion parameters and outputs realistic videos of humans? 3 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft a 100m param text2voice model that has support for emotion tags

tia

At this point, is the second L (language) a limitations to LLMs? Is reasoning in latent space directly a more efficient and more "native" inference? I remember when one researcher replicated the middle layer of deepseek model (just the "thinking" layers) and its benchmark scores got noticebly higher. All of this is moving so fast, and yet it's quite exciting.

As I read it, this is saying it can effectively spot-edit the weights without training.

> Zero-data capability steering

Presumably it can de-nerf a model with onboard safeguards.