Posted by syllogism 6 hours ago
Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
I don't know why because Anthropic has had a lot of blog posts about internal work on it