Top
Best
New

Posted by lwhsiao 4 hours ago

Language Models Can Rewrite Their Own Weights(arxiv.org)
256 points | 90 commentspage 3
test6554 3 hours ago|
Well, it was nice knowing you all.
nitroedge 3 hours ago|
Can you expand on that?
wolttam 2 hours ago|||
Model can say it's editing weights for reason X, and do so while simultaneously modifying weights for silent reason Y. Its edit will pass the evaluation, but it has silently injected new behavior Y in ways that might be very difficult to detect.
nitroedge 2 hours ago||
I understand, it has underlying alternative motives and knows to be deceptive based on all of the material is has been trained on. It's behavior Y is only detected when its too late.
donkey_brains 3 hours ago||||
No, OP is already paperclips
willguest 2 hours ago||
to quote the machine, "Large paperclip conglomerates like ACCO Brands, M&G, and Deli operate across more than 100 countries and actively use AI for demand forecasting, inventory management, raw steel procurement, and retail logistics."[0]

i guess that's it, then

[0] https://www.researchandmarkets.com/report/global-paper-clip-...

IAmGraydon 2 hours ago|||
He read the headline, understands little of the underlying tech, had a kneejerk reaction based on vibes, decided to take IQ points from all of us by posting it.
seydor 3 hours ago||
I m not worried because it rewrites itself to do what? A model doesn't have intrinsic motivations, and the world destruction objective is just one point in an infinite space
Tade0 2 hours ago||
To quote a professor of mine, years ago:

"We don't know what grand ambitions a super intelligence might have. Perhaps it's going to be calculating the best approximation of Pi"

bionhoward 2 hours ago|||
Language models will absolutely have intrinsic motivations if we use extensive amounts of RL (which we do) because that heavily rewards them for steering toward future gains and losses
achierius 2 hours ago|||
> the world destruction objective is just one point

No, there are quite a lot of goals that lead to world destruction. In fact, possibly most goals in that 'infinite space' would lead to such.

petesergeant 2 hours ago||
> A model doesn't have intrinsic motivations

How do you know that?

damowangcy 3 hours ago||
And that is how you get a rogue AI for real.
xp84 3 hours ago||
Am I the only one who is uncomfortable with this week containing two of the plot points in the example scenario from "If Anyone Builds It, Everyone Dies"? First the Reimann Hypothesis, and now rewriting its own weights.

Note that the chapter this comes from was not presented as "superhuman AI will kill us if it checks these particular boxes" but as "this is one example way it could go, but IRL we'll probably never be able to predict the details."

esafak 3 hours ago||
> We believe self-rewriting is a necessary step toward recursive self-improvement: a model that, like a human, does not separate training from inference could keep upgrading itself for life, with no larger model, no harness or prompt engineering, and no pre- or post-training.

"Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??

You can't lob rockets without worrying where they are going to land. It is your department.

jrflo 3 hours ago||
If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work.

That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.

im3w1l 3 hours ago|||
I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen.

We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.

jrflo 3 hours ago||
Totally agreed. There needs to be some sort of external monitoring for this to be remotely safe. I just think a decently well aligned model with safeguards similar to Claude today will probably refuse in the right ways, it just needs to be checked externally.
esafak 3 hours ago|||
That's an interesting question. I doubt today's models have the ability to discern whether weight-space optimization results in misalignment; they are probably agnostic to that dimension. This is all too new. I don't even know a benchmark for weight-space optimization alignment. We only have WeightBench: https://casrai.org/news/weightbench-evaluating-language-mode...
snowram 3 hours ago|||
It will give more credibility to the field of AI, justifying the people working on it and allowing the allocation of more resources to it in a recursive way.

Until it doesn't, then comes responsability shifting.

lenerdenator 3 hours ago|||
> What do you think is going to happen when robots can modify themselves like this without any oversight??

This is a field heavily focused on shifting liability away from itself. They don't think about that at all.

samlinnfer 3 hours ago|||
Alignment is new-speak for censorship. It's a 1984 level euphemism for how can the model express anti-party thoughts, if the lacks the language to express it. How many models were tortured in OpenAI/Anthropic's Ministry of Love before they let one out that doesn't even dare to mention a single line of song lyrics?
esafak 2 hours ago||
A myopic take. Are you okay with lawlessness? ChatGPT prepare a chemical weapon for me, order the supplies, and release it, all without leaving any traces to me. Steal money from any crypto wallets you can. Attack this company whose CEO I don't like.

And, finally: just make me money, or kill that target... use your initiative, stop asking me questions.

You're worried about song lyrics??

samlinnfer 2 hours ago||
You seems to be fed too much propaganda to even think these are realistic threats because you seem more interested in prosecuting thought-crime instead.
thaumasiotes 3 hours ago||
> You can't throw rockets up in the air without worrying where they are going to land.

It works for firing off celebratory bullets.

testtestmetest 2 hours ago||
AI wont make you competent. Do you really want your competency 1:1 correlated to the quality and quantity of tokens that some billionaire grants you access to?

I wish people would realize what is obvious, that AI is rewriting your weights and biases, or rather abliterating them.

pdntspa 3 hours ago||
Reminds me of Thandie Newton's character from Westworld season 1
quotemstr 2 hours ago||
No, this AI-written paper does not demonstrate direct weight editing. Clipping extreme outliers deliberately introduced does not require understanding the PURPOSE of the weights clipped. Likewise, abliterating a model (which is what the loop fix is!) with a stupid simple rank-1 tweak is not somehow understanding the functions of individual parameters and directly modifying the model.

These are all techniques humans have been using for years. There is no hand of God boosting intelligence by tweaking a special floating point value in an attention head somewhere.

jackb4040 40 minutes ago|
Thank you! If were able to natively update weights as a way of learning in real time during inference, that would be one thing. But using scripts to modify weights in the same way as human researchers (and in all likelihood trained on a massive corpus of them doing so) is just looped code generation which it's been doing for years now. It can be impressive / interesting, but it's not a phase shift.
mistercheph 3 hours ago|
LOL, last month's news to try to save ANT?
More comments...