I didn't think I'd be using Astra to mess around with Qwen weights today, but here I am. I asked it to remove Chinese, 532.5 MB → 483.2 MB.

Great, instead of backpropagation, we now have slopagation.

almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest

Someone familiar with this stuff able to enlighten me how this technique differs from this 2024 research? Refusal in Language Models Is Mediated by a Single Direction: https://arxiv.org/html/2406.11717v3

Like micro surgery, opening up the LLM safetensors file and performing micro text edits via small scripts?

They are slicing, probing, and running linear algebra on the tensors, observing the output in the console, then writing the edited weights back to disk, and stitching it all back up.

Johnny Mnemonic level

Seems to be mostly around the input or the output layer not the deep bowels of the nets. Near the input layers, neural nets basically do template matching, so recognizing the paddle, the circle etc is not that surprising. The innards is where it gets more interesting and that's not so clear whether Astra can interpret.

This reminds me of the Interpretablity blog from Anthropic where they used their model to read the weights from the same model itself, interpret it, and tune it. I had a similar idea with what is outlined here, but I didn't have the resources to try this. I did ask several AI models about the idea and they said it wasn't going to work.

Research questions: Can model read and write weights of the same sized model (its own weights), and of a larger model?

This is mind blowing.

This opens so many avenues on using Astra or any large model to one shot smaller purpose built models for Agents, Imagine Agents writing or editing smaller weight models when doing agentic loop.

The implications are immense if all of this is generalizable

I just want to say, absolutely incredible work by the authors! And the presentation is also very well done. Kudos and congratulations!