But i also get the immediate "but how is anyone supposed to run this locally" feeling
Maybe we need a new network. Like, Nextdoor + counterstrike pub servers, but for local LLM. We can call it semilocal
:)
Creamsicle47 19 hours ago|
[flagged]
yukim198 13 hours ago||
[dead]
kmaii 18 hours ago||
[dead]
tgtweak 20 hours ago|
If someone released a 0.1bit quant I can run it. /s
user_7832 19 hours ago|
Fwiw, the Qwen 3.8 27B at 4 bit quant is apparently quite good and is probably the best recent model. It runs (iirc) on 18-22ish GB of memory.
ricardobeat 16 hours ago|||
I read somewhere that Qwen 4 is right around the corner. Praise be!
robertkarl 19 hours ago|||
quantization has come a really long way. Q3 is fine, especially GSQ-RCO. You can run 27B well on 16GB of VRAM.
zozbot234 18 hours ago|||
The point of quantizing larger models is to beat other models running on the same limited-VRAM platform, i.e. 16GB VRAM in your example. You're doing "well" compared to other models meant for running on 16GB VRAM; you're not doing "well" at all compared to those running higher precisions for the same params up to at least Q8. (16 bit and higher is where the gain from further precision gets real iffy.)
adornKey 17 hours ago|||
With e.g. Strata you can run 125B Q4 very well on 16 GB of VRAM. 27B already has vintage vibes... OH, the good old 27B days...