This is a meaningless metric without knowing how long the sex takes

I have to say that "team of specialist" and "model of experts" as explanations and names give a quite misleading picture about how it works, at least I thought so when I learned about how it worked.

À 2 bit quant will (at best) get you about 80% of the full models memories. That's from a purely information theoretical sense. IRL it's worse than that.

Capability can still be better than 80%, but that depends on extensive post-quant recovery training to essentially rebuild the models internal manifold to route around the damage.

So, 3.8 Flash Next is better than GLM 5.3 for some things. This version is not.

FP4 is as low as you want to go if you want to retain most function and recall. Below that the noise gets too high and information becomes unretrievable. If it's a full quant you don't get to choose which info is lost. Just 20% randomly.

One interesting thing about this model is that it uses engrams. Meaning you can separate much of the storage from the compute and quantize them differently. That's not what they did here though. Here it was indiscriminate.

Terribly

Yes, I am running the same on a 64G mc. It's good, but slow at 25 tps on average! I want > 100 tps - but I don't have $5k to spare for an m5 ultra or an nvidia setup.

So I still wonder if one could get good enough quality with a faster higher quant or superoptimized Qwen3.8-27b with dflash2

https://huggingface.co/nathansutton/Qwen3.8-27B-Ternary-Bons...

or a MoE retrofit like Qwen3.8-35B-A3B with or without mtp

https://huggingface.co/NovaeonStudio/Qwen3.8-35B-A3B-Distill...

https://huggingface.co/IsValorum/Qwen3.8-35B-A3B-Distill-MLX...

I haven't seen much in the way of benchmarks of those smaller quants. How does it compare to e.g. various generations of Opus?

There are options. If you have fast CPU RAM (ddr5) and PCIe bus you can run Qwen3.6-35b-a3b at good speeds+~100k context (I have a friend who is running this setup). If not, you're stuck with the much smaller models: Ling-3.0-tiny, Spark-X2.5-4B, LFM2.5 in 8b-a1b or 2.6b, and FrogNano-4B-2609 looks promising. But 6GB is a tough squeeze, 8gb is a lot cleaner, and if you have 12gb you can run strata like OP.

Interested to know throughput on 7900 XTX and what setup you're using?

Yeah, -10% accuracy (probably more like -20% in reality) sucks, but only if you could be running it at 100%.

That's the exciting part of this - before the best you could run on <24gb vram was qwen3.8-27b at q4 quantization. Now you can run a nerfed 125B parameter model on under $800 of hardware, and it beats a less-nerfed 27b model.

(64GB not VRAM, I meant) I also see people claiming fast performance on a 128GB machine, which is not exactly consumer hardware)

I wanted to know as well. KV cache is Q8 by default but can be set up at full precision in the config.