what I am curious about is - what is more expensive, the GPU compute of a traditional fine-tuning loop, or the LLM tokens spent trying to lockpick together a solution. It's impresive - but it definitely feels like it takes a lot of luck and brute forcing with a ton of tokens. Nevertheless, if we are able to train tiny models to be really good at weight-space programming, then the calculus may change.

I love the idea that this can be used for optimizing inference. Other than that, this reads more like "a new distillation technique is discovered".

I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.

This looks very similar to that in a way.

I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.

Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?

AI should be able to do a better work than gradient descent. Not by guessing the final weights, but by gradually, iteratively adjusting the weights

Actually, I'm skeptical how useful this is. Intuitive speculation: Astra can only improve weights to its own level, so can't train itself via this method. Even "continuous learning" this way may be no more effective than putting a summary of recent news in the prompt, with the same kind of decay, as "update Astra's knowledge while keeping performance" is much harder than "teach Qwen to do basic math".

On the other hand, already OpenAI and Anthropic heavily use their LLMs for research and coding, and it's probably not only much faster but smarter (remember, an LLM solved a millennium problem; more relevant, LLMs excel at combining disparate deep research concepts in ways that humans can't, simply because no-one studied those disparate fields). And the "singularity" is already here if you consider capitalism and society to be collective intelligences: the biggest obstacle to an AI pause, "we cannot get everyone to collaborate without defecting", is very strong.

We have AGI by most metrics and are getting closer and closer in the remaining ones. I predict improvement won't be exponential, and there will never be a day when AI perfectly replicates humans; rather it will be many small steps, getting closer to human in some areas while greatly surpassing in others. But still, even today it's actually very useful.

Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).

I wonder how it would do with distilling models by hand?

Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next

Is there any chance they’ve just seen these weights before and are recognising them?

If not this is mind blowing.

However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.

Interesting follow up would be to see if any open weight models can do this and then run some interpretability experiments on them to figure out which motifs and circuits light up

I guess this makes it extremely easy to add vulnerabilities