Posted by syllogism 10 hours ago
The author is being equally disingenuous.
That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.
The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.
The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.
AI coding assistants make it much easier to conduct experiments and improve AI models, which in turn improves the coding assistants etc.
Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).
[1]: https://aclrollingreview.org/reviewerguidelines#-task-3-writ...
Trying to find every single angle and option to break something is how we verify the truth of something.
If something is objectively true, it will survive the scrutiny. If it does not survive, it wasn't objectively true.
This is how science works. That ain't no flaw. It is a flaw to be offended by science doing science things.
And the examples in the OP's blog post just go to show that reading comprehension is at an all time low:
> Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
The table just below shows this statement to be *entirely* factual.
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
Any further implicit assumptions are on the reader, not the writer, as they never stated "that the Qwen experiment worked ‘just by looking at the raw weights’". You can certainly point out issues in the experimental design, but just stating you dislike the set of experiments and is yet another poor take that directly contradicts ACL's guidance for reviewing, exactly because it does not advance science.
I believe the post definitely plays off the implication that the Qwen experiments are connected to the experiments that read the raw weights and have no Python harness, when this isn't the case. There are experiments where Astra reads weights directly for very simple models, and there are experiments where Astra performs ML research tasks by analysing the problem and writing Python code. The authors imply that the same technique underlies these but that isn't the case.