Posted by lwhsiao 4 hours ago
i guess that's it, then
[0] https://www.researchandmarkets.com/report/global-paper-clip-...
"We don't know what grand ambitions a super intelligence might have. Perhaps it's going to be calculating the best approximation of Pi"
No, there are quite a lot of goals that lead to world destruction. In fact, possibly most goals in that 'infinite space' would lead to such.
How do you know that?
Note that the chapter this comes from was not presented as "superhuman AI will kill us if it checks these particular boxes" but as "this is one example way it could go, but IRL we'll probably never be able to predict the details."
"Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??
You can't lob rockets without worrying where they are going to land. It is your department.
That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.
We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.
Until it doesn't, then comes responsability shifting.
This is a field heavily focused on shifting liability away from itself. They don't think about that at all.
And, finally: just make me money, or kill that target... use your initiative, stop asking me questions.
You're worried about song lyrics??
It works for firing off celebratory bullets.
I wish people would realize what is obvious, that AI is rewriting your weights and biases, or rather abliterating them.
These are all techniques humans have been using for years. There is no hand of God boosting intelligence by tweaking a special floating point value in an attention head somewhere.