I always wonder is it safe to run one of these models on a personal pc, or do you guys recommend something like docker, sorry a bit new to all of this.

Note that these quants are not quantized uniformly, so 4-bit isn't actually a "true" 4-bit here, so these observations won't necessarily hold up to other quants which might be done differently.

I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM. It run slower than molasses at 2 too/s kind of thing. Not sure it is usable at that rate for anything.

Could this be used to estimate how many fingers LLM have?

1-bit, 2-bit, penny and dime.

It's strange that the author has completely ignored the 3 bit quants which allow someone with a 16GB GPU to have 100-120k and still get full performance. You can't run any of the 4-bit quants on a 16GB gpu with enough context to be useful for all but the most basic tasks.

General purpose agents can need up to 30k just to reply with "1+1=2" because their prompting is so overloaded. 60-70k is decently usable, still not great for anything complex. A long running task in a general purpose agent can easily hit 100k.

What the vast majority of people care about is performance around what desktop consumer GPUs can run. 8GB, 10, 12, and 16GB of VRAM. What do models that will run at full performance, do?

Also important to know is how Qwen3.8-27B stacks up against qwen3.6-35B-A3B, which due to being MoE, will run on a 16GB card with plenty of speed 90% of the time, at higher quant - so you get more parameters and better quant. But 3.8 is supposed to be "better", so...?

Those benchmarks are very interesting.

But is there any model that actually works in a decent way at quantization of 1?

Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.

Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work that gets otherwise rejected.