Top
Best
New

Posted by stared 20 hours ago

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses(quesma.com)
253 points | 124 commentspage 3
sanjusangh 15 hours ago|
Isko ek karna hai
quietraster 19 hours ago||
the 4-bit matching bf16 on terminal-bench is a useful data
zrail 19 hours ago||
I've been running Unsloth IQ3_S on my 5060ti with mmproj offloaded, getting 600-1000 prefill and 30-50 tg with this config:

       /data/llm/llama.cpp/build/bin/llama-server
        --threads 4
        --threads-batch 8
        --batch-size 4096
        --ubatch-size 256
        --port 9999
        --temp "1.0"
        --top-p "0.95"
        --top-k "20"
        --min-p "0.0"
        --presence-penalty "0.0"
        --reasoning auto
        --reasoning-preserve
        --reasoning-budget 4096
        --gpu-layers-draft all
        --spec-type draft-mtp,ngram-map-k4v,ngram-mod
        --spec-draft-n-max 3
        --spec-draft-p-min 0.75
        --spec-ngram-mod-n-match 24
        --spec-ngram-mod-n-min 4
        --spec-ngram-mod-n-max 16
        --spec-ngram-map-k4v-size-n 8
        --spec-ngram-map-k4v-size-m 16
        --spec-ngram-map-k4v-min-hits 1
        --n-gpu-layers all
        --ctx-size 131072
        --repeat-penalty 1.0
        --jinja
        --metrics
        --model /data/llm/models/unsloth/Qwen3.8-27B-UD-IQ3_S.gguf
        --chat-template-file /data/llm/models/qwen3.6-chat-template.jinja
        --fit off
        --flash-attn on
        --cors-origins localhost
        --mmproj /data/llm/models/unsloth/Qwen3.8/mmproj-BF16.gguf
        --no-mmproj-offload
        --parallel 1
        --kv-unified
        --cache-type-k q4_0
        --cache-type-v q4_0
        --cache-type-k-draft q4_0
        --cache-type-v-draft q4_0
zrail 17 hours ago|
Too late to edit, but a few other things to note: I minmaxed the draft config. On my typical coding workloads it gets around 70% acceptance, more variable on prose.

The chat template is froggeric's fixed qwen template, v22.5 as of today.

Neat_comfort007 6 hours ago||
[flagged]
hefu_hk 8 hours ago||
[flagged]
paidx 10 hours ago||
[flagged]
InvectusXIV 19 hours ago||
[flagged]
john_rood 18 hours ago||
[flagged]
dotinvictim 18 hours ago|
local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable
kennywinker 17 hours ago||
It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).
TomBombadildoze 17 hours ago|||
Beats Opus 4.6 at what exactly? It certainly isn't code.

I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.

kennywinker 17 hours ago|||
https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-o...

Index methodologies here: https://artificialanalysis.ai/evaluations/artificial-analysi...

Also see some specific benchmarks here: https://huggingface.co/Qwen/Qwen3.8-27B e.g. qwen scores 61.7 on swe bench pro, while opus 4.6 scores 53.4.

If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.

brandon272 14 hours ago|||
What configuration? What harness? These matter greatly to how a local model performs, in my experience.
fshr 14 hours ago|||
> 5060

I think you're referring to a 5060Ti 16GB, yes?

32k context is easily done there. 64k can work with a more aggressive quant, but you lose a bit of speed.

kennywinker 14 hours ago||
Yeah 16gb. For longer context, a 3bit quant is needed. Yes it’s tight on a 16gb card - can’t wait for the bubble to pop so hardware prices fall.

But I don’t quite follow you - how does a more aggressive quant slow it down? Less bits per token means faster inference not slower.

Winfred-zz 13 hours ago||
In my own experience, qwen3.8-27b 4bit can consistently find bugs in software written by sonnet 5 and opus 5. But it does do that at maybe 1/10th the speed. Still a pretty good deal if you're coding without wanting to spend big.

qwen3.8-27b 4bit has a following specifically for being exceptionally gifted for such a small model.

noir_lord 10 hours ago||
It’s genuinely the first local model that has actually made me keep it around, it’s pretty good at spotting things existing tooling can’t, like comment rot/drift/summarising accurate git changes, it’s also the first local model that pushes back when it thinks it is right (I don’t dislike that even when it is wrong, it still hallucinates things that don’t actually work the way it confidently states they do).

It does clear the useful enough to be worth it bar though.

Zero interest in remote models but local ones if they offer utility, sure.

Runs pretty well on a 7900XTX as well.