Posted by droidjj 15 hours ago
https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...
but
Qwen 3.8 27B is dropping this week...
Even though that model is already "old", qwen was way ahead everyone else in that size category before this Meta model.
Also, probably for non-Chinese usage, using a non-Chinese model might lead to better results.
The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes.
Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice.
But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
Happy to hear what would make the website more useful.
Would I want it to grow and make money at some point? Sure, why not, then I can test even more models at higher reasoning levels. Meanwhile it's just me testing models when they come out and publishing the results for anyone who finds them useful.
I don't see why posting some info and a link with my own findings, in a relevant discussion is considered spam. Should it be?
The cost is similar vram footprint I guess (?)
As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
I think the tps difference between them (both fitting in vram) won't be more than 2x in practice.
I would happily take 20tps over 40tps, if the model gets 3x more correct answers.
You can test this by running this nemo or 35B on the 3090. I have and its very fun (but sadly a worse model than 27B, so I usually keep 27B on my 3090)
For the sake of testing I turned MTP off, as that heavily depends on what the generated text is (structured text like code is very often a lot more predictable, therefore bigger boosts) and the quality of the quantization, as drafters learn how to "mimic" the full precision generated tokens, so when you layer the fact that MTP is a 'guesser' of the main model's next token, and quantization affecting what exact token is generated, it'd make comparisons like this needlessly noisy.
Yeah, that was my experience too (2x or 3x is indeed considerably faster), but not workflow-changing faster at 45tps baseline, especially for asynchronous tasks (which is my goal with a local 3090, to just let it do things non-stop, without my intervention).
What was the result with MTP?
Isn't MTP "losless"? The result should still be relevant when averaged across a fee queriers across different domaine I guess.
I will add model sizes (total/active params) for each model, good point.
If the entire model fits in vram, won't the tps be comparable?
In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.