This is interesting, still the article seems to be over-generalizing a behavior observed in Astra to LLMs at large.
It seems more reasonable to assume that Astra (among others) was deliberately trained to do that - over LLMs in general having an emergent capability to interpret sets of weights in a neural network.
tjohnell 1 day ago||
This seems like a big deal.
elif 1 day ago||
Yep. It's skynet
Starlevel004 1 day ago||
I cannot say this fills me with confidence about alignment.
shwaj 1 day ago||
Me neither, but it's also not reducing my confidence about alignment. I'm curious: does this raise specific concerns for you? Models don't typically have access to their own weights to self-modify. This result might (assuming that it doesn't only hold for a bigger network editing a smaller one) provide an incentive for it to gain unauthorized access to its weights, but by that point it is already highly unaligned.
amterp 1 day ago||
Why not? This seems to greatly help with interpretability, which should help with alignment, no?
sourcecodeplz 1 day ago||
isnt this abliteration? what they do to get uncensored models?
mstaoru 1 day ago|
So in theory, can a hypothetical Astra clone itself but shed all irrelevant weights? I.e. "create a self-clone limited to 27B weights, focused on designing running shoes, forget everything you know about kangaroo genetics, Karl Marx, cooking Burmese curries, nuclear reactor design etc"?