and this is the pelican from the iq4_xs model https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...
NVFP4 quant
It's new arch demo for future Qwen 4 family, but (as I understand) training recipe/data is same as any other 3.8 model.
How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation.
Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "6b per token" network compared to the dense capabilities...
Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes.
Modern LLM's are wildly inefficient.
I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.
It's not world changing, but at those speeds I'll take anything I can get.
(The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).
Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk.
Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?
Kinda wish I had a Strix Halo here to play with as well.
But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)
I got prefill up to 190 tok/sec just now, BTW.
Qwen 3.8 flash: $0.16 / $0.47
Compared to
Deepseek 0723: $0.03 / $0.075
(units in USD/m tok)
8t/s though apparently and their cache hit rate is terrible so I don't think it's worth it over Relace.
DSV4 Flash 304B params, 167 GB download (at full size)
Qwen3.8 Flash Next 180B params, 360 GB download (at full size)
This branch works now: https://github.com/unslothai/llama.cpp/tree/qwen4exp/qwen3.8...
cmake -B build -DGGML_CUDA=ON cmake -B build -DGGML_METAL=ON cmake --build build --config Release -j --target llama-server llama-cliWith 2x Sparks, I am getting 40 t/s. I'd guess that without MTP you'd get 12-15 on 1 Spark, maybe 20 with MTP?