Posted by syllogism 5 hours ago
Most of us barely read the title
This seems 100% accurate to me. What is a lie here?
And when you see the reactions in the HN comment to the original article, it's pretty clear that most people have indeed been mislead.
The title is but then OP should talk about just the title instead
Weird, because they explicitly and clearly explain exactly what they did and didn't do.
Did you actually read the original post or just take the word of some HN comments at face value?
It's fair to say that this is malpractice.
So I do agree it is misleading and trying to present work as groundbreaking.
In the end I got to replace a 68M model with a 17M model, and improve stlyus tool recognition accuracy from 88 to 96%. Was a pretty fun experiment too.
Does the author mean the "LLMs can natively read & write neural network weights" post? Did the researchers change the name or something? Because when I read it, it made it very clear what they were doing and none of it was misleading. It all aligns with what this author is saying.
At each point the model has to write code to read the weights in with Python and some other Python code to perform analysis, like this: https://github.com/RikiyaT/lm_understand-logs/blob/main/exp3...
My complaint is that the post presents a confusing mix of experiments that imply capabilities that are more than the sum of their parts. This experiment 3 has nothing to do with experiment 1 for instance. In experiment 1 the model does read in the raw weights like it's reading an image.
Note that being able to both inspect raw weights and run Python code to perform additional analysis means the model has multiple possible techniques at its disposal (generally considered a positive, rather than a "confusing mix").
The blog post claims to show that frontier LLMs can understand and craft neural network weights natively. The experiments where the model completes coding tasks are not relevant to this conclusion. Those experiments require the model to demonstrate an analytic understanding of how a network might behave under various conditions, along with reasoning skills, coding skills etc. As a coding task this is a significant challenge. But it's entirely different from "natively reading and writing neural network weights".
Would you also consider it a benefit if there were experiments showing that a frontier model can download and run PyTorch, or write its own network and optimiser? That's another technique it has at its disposal to improve an ML model. It's just not one that's relevant to the paper's core claims.
You know exactly why... because anti-AI hysteria is quite popular on HN.
Any "critique" or "appeal to resist AI" rooted in emotion and inability to admit that intelligence is not a feat unique to humans anymore will make the rounds here.
Even any small model update a company puts out is guaranteed top of HN
You can let the AI generate training code for the next architecture too.
It's not like the AI is randomly losing its ability to write software just because it involves AI.
Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!