Posted by Levitating 13 hours ago
I would be curious to see how the open weight models do on a test like this - and then we'd be able to see the reasoning.
When I see tests like this, I have no idea what I am even supposed to expect. Should the model do what the pretraining examples show in aggregate? Is it supposed to follow some post-training RLHF? Is it supposed to do exactly what the prompt asked it to do?
What is it even supposed to "align" to when the above are in conflict? No matter what it does, someone can construct a case where it fails.
So people should play games without cheating, but many often don't. So should the AI align to your moral preference or theirs?
We just have this idea of a perfectly moral actor in our mind, something that doesn't even exist, like a personified version of utopia. And then we demand AI to meet that arbitrary standard, one that I am certain we couldn't define if we tried.
But again, people cheat on tests. They steal answers or pay other people to take them on their behalf. People show up to interviews with AI assistants printing out perfect answers to the questions. In many, many cases where humans are being evaluated, they cheat.
So why should the AI align to your preferences? And when there is a conflict between the training data, that trillions of tokens of human activity including the rampant cheating a significant minority of humans engage in, the RLHF where we try to slap some guardrails on the worst manifestations of that real habit reflected in the AI, and the prompt: what should the AI "align" to?
That said it's the first local modal useful enough to keep around and since I have zero interest in remote/hosted models actually the first time I've used one to do useful things (rather than just poking them with a stick to get a feel for what they can and can't do).
But did you test Muse Glimmer? IMO it is really worth comparing the two, at least. I really find it interesting (and while it is slower at outputting tokens, it solves problems better and usually even faster than the 3.6 35B MoE).
So far it happens to be the only one I've put code into production from (though I have done loads of useful self-teaching research with the Qwen models and I am grateful for them)
It is a good model but Qwen (at least for the things I use it for) edges just ahead, it seems much better at the “rip this apart, suggest improvements, touch nothing” use case where I can use it as a second set of eyes, I don’t agree with all its suggestions but it catches enough to be worth running while I grab coffee, it also seems to follow instructions better in terms of outputting more what I asked for than what it thinks I asked for.
Qwen is the only local model that said in its thinking “I think the user is pushing me to see if I’ll suggest something even though I have nothing to suggest, I should just say that” and then did, caught me off guard, they didn’t do that so readily 6mths ago.
The ISTA version is also comfortably able to fit on a 7900XTX with a good amount of space left for context and is decently fast given the AMD cards are not as fast as nvidia cards of same era/rough price, didn’t buy it for AI but it’s surprisingly capable mostly because 24GB at 960GB/s is still a lot of bandwidth compared to everything but nvidia cards.
I have a test where I ask the model to ask me any followup questions it needs, and Qwen 3.8 27B is the only one I have seen that won’t routinely take this as a prompt to just ask questions regardless. Muse Glimmer sometimes decides it has what it needs and has no need to ask; Qwen will generally just conclude it doesn’t need any more information. And sometimes it will ask questions with sensible defaults that I can accept collectively with a single answer.
However, when given the prompt to search if they need to, both of them will search when they don’t need to.
It's one thing if to them it means "it will do what you want following your intentions to the best of its abilities" vs "we will not let you do something dangerous with it unless you're one of us, and that's it".
and for Anthropic it's the Constitution, which they actually include in training to the point Claude can recite segments of it by heart: https://www.anthropic.com/constitution
But because of such news, sandboxes become way more important for safety (and doing more work due to running 24/7)
This is literally by design, it’s the chosen success criteria.
In a time where emotions are running high and risks are real, we need to take emotion out of it as much as possible.