Unsloth Dynamic 3.0 GGUFs

Posted by jonesy827 16 hours ago

281 points | 99 comments

walrus01 12 hours ago|

It would be nice if unsloth published GGUFs would use a version number or something, because now I have multiple different files on local storage that otherwise have exactly the same name.

"Qwen3.8-27B-UD-Q8_K_XL.gguf" for instance.

The one downloaded at least 4 days ago is a different thing and is NOT the "Dynamic 3.0" GGUF which I am now downloading, which I presume will have a different sha256 checksum?

The unsloth page says dynamic 3.0 is released "today", but I have an older copy of qwen3.8 27B Q8 which I downloaded, if I remember right, at least 4-5 days ago...

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

prima-facie 10 hours ago||

You can manage these easily with the huggingface python app.

hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf

and then see them with `hf cache ls`.

Prune old versions with `hf cache prune`.

14u2c 7 hours ago|||

Using git is another option, you'll get the actual commits/history to reference:

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

cd Qwen3.8-27B-GGUF

git lfs pull --include="Qwen3.8-27B-UD-Q8_K_XL.gguf" --exclude=""

hedora 9 hours ago|||

That's terrible. My computer shipped with a filesystem and curl.

cicko 4 hours ago|||

You can't run models with filesystem and curl.

embedding-shape 1 hour ago||

To be fair, you can't run models with "hf download" either...

mycall 9 hours ago|||

There is always SHA256 as the alternative.

jlbprof 11 hours ago|||

How can I tell which one I have? I dl'd mine a few days ago.

walrus01 11 hours ago||

I have just run a sha256 checksum on both copies now, the one I downloaded 4-5 days ago, and the one that's on the unsloth huggingface page released today, and will see if they're the same or different.

(downloaded in the last few hours after the announcement of dynamic 3.0)

Qwen3.8-27B-UD-Q8_K_XL-unsloth-dynamic3.0$ openssl dgst -sha256 *.gguf

SHA2-256(Qwen3.8-27B-UD-Q8_K_XL.gguf)= af36ecb6b5db1407953345b746c14ac93f0657dda413910b4348683a2d990377

=====separator=========

downloaded at least 4 days ago:

Qwen3.8-27B-UD-Q8_K_XL-unsloth-original$ openssl dgst -sha256 Qwen3.8-27B-UD-Q8_K_XL.gguf

SHA2-256(Qwen3.8-27B-UD-Q8_K_XL.gguf)= af36ecb6b5db1407953345b746c14ac93f0657dda413910b4348683a2d990377

So they're actually the same thing, but the announcement says released today... Please let's not confuse the end users any more than they already are.

CMay 10 hours ago||

I think they said they were keeping the old UD 2.0 quant for the larger sizes? So maybe they kept those the same and simply reuploaded them. They said the newer UD 3.0 quant performed worse on some things for the higher quants. So now it's a mix of UD 3.0 and UD 2.0.

m463 11 hours ago||

I wonder if there should be some kind of header sort of like a README inside mixed with something like EXIF information that camera images embed.

walrus01 11 hours ago||

there's metadata in a GGUF (as can be seen when loading one into llama-server with verbose level 4) but it doesn't appear to show what the expected sha256 of the file is.

gunalx 11 hours ago||

Well that would not work as the metadata is also part of the sha, and if you have a sha predictably contain or hash to itself you have a bad hashing algorithm. (Unless of course you mean sha of the weigths and not the gguf itself)

nagaiaida 8 hours ago||

you can just hash the metadata masking out the hash as all zeroes or anything similar to get around that, like header checksums that ostensibly include themselves

Alephinitesimal 12 hours ago||

I mostly use local models when the data has personal information. Earlier this year, I felt the coding quality was still not as good as Claude Code.

One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data.

This way the real data never leaves my machine, but I can still use a stronger model for most of the coding.

latentsea 9 hours ago||

Qwen3.8-27B has been the turning point for me. It's not as strong as the absolute frontier, but it's the first time I feel local coding models are actually functionally useable as daily drivers.

Forgeties79 8 hours ago||

Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster

Alephinitesimal 4 hours ago||

That's interesting since both models are dense. I wonder if this is more of an optimization issue with 3.8 rather than something inherent to the architecture.

johnnyApplePRNG 4 hours ago||

Could have sworn I read these were the same architectures the other day .... 3.6 and 3.8 at this size.

illusive4080 8 hours ago||

Have you tried on personal finance analysis? That is what I most want to do but haven’t gotten around to it.

Alephinitesimal 4 hours ago||

I did try some finance analysis earlier this year. I was using a DGX Spark, so I could run some relatively large models, but the results were pretty mixed at the time. I honestly can't remember which models I used anymore.

Might be worth trying again now though.

jacquesm 1 hour ago||

I'm having a really hard time doing on twin DGX spark what I could do on my quad 3090 rig (which is a scaled down version of what I was using before, the power requirements and the noise were really an issue but I loved the speed and the amount of VRAM). The results tend to be inconsistent, there is lots of looping, far more tokens generated for the same job and lower quality output. I suspect there is some kind of regression in the B12X kernels or something to that effect because none of that should happen, the exact same model on both machines gives wildly different results. Probably this will sort itself out over time. If I may ask, what model / software combo were you using?

xlayn 15 hours ago||

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.

gruturo 13 hours ago||

The reason for running those insanely low quants is to fit in extremely limited memory budgets. The first thing you sacrifice is speed, then context and accuracy (up to you in which order). IQ2_XXS and below is desperate/proof of concept territory. If you have a spare half gig for the MTP drafter, run a larger quant instead, it will be less incoherent, and damn the speed, it won't be garbage at least. Only around Q4 I'd allocate the comparative luxury of more memory for a speed increase. At least on a dense model. MTP makes a lot more sense (but helps statistically a bit less) on an MoE.

Qwaiting for that 3.8-35B-A3B

danielhanchen 12 hours ago|||

Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed.

As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that.

But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL

xlayn 12 hours ago||

Daniel, question I got the Qwen3.8-27B-UD-Q2_K_XL.gguf from https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_in... and continue with my testing, but the model quickly felt into a loop of asking the same thing over and over again, I have seen the MOE do that but not the dense ones.

And I had similar experiences when Qwen3.8-27B unsloth images just came out with the full Q8_K_XL, I'm using an AMD setup which has modifications to save to disk the kv, but your (assuming you are part of the unsloth team) for some reason have been giving me similar issues.

I tried https://huggingface.co/mradermacher/Qwen3.8-27B-Uncensored-G... the 8 bit, 6 and 2 bit... the 2 bit almost use the complete KV doing it's thing and didn't loop itself.

It can be something in my setup, there is a very high chance of that, but the previous 3.6 images from qwen, the 27B, the 31A3 and 122 they are all unsloth and did work on my setup without issues...

Again could be my setup... let me know if there is any data I can supply to you to debug if needed.

kamranjon 6 hours ago||

Are you using the recommended settings for temperature and such? https://unsloth.ai/docs/models/qwen3.8#recommended-settings

Often times I run into issues like this it’s because I am using settings for a different model or just forget to set them up.

mike-the-brain 15 hours ago|||

you can still have it, no?

> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed

xlayn 15 hours ago||

my bad, you are totally right, thanks!

walrus01 12 hours ago||

Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.

zenoprax 9 hours ago||

Sure, but this is true for all lossy compression (audio, images, etc.)

Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3

walrus01 5 hours ago||

probably the best experience would be deepseek v4 flash 0731 (it takes about 170GB RAM on the server side for the full thing and RAM reserved for 1M context) via opencode's $10 a month plan until you use that up, it's either Q8 or full precision. Assuming you're ok with doing things with external inference.

kennywinker 2 hours ago||

Why would they have listed how much vram they had if they were looking to rent gpu time on someone else's machine?

walrus01 2 hours ago||

A casual review of my comment history would show that I've been nothing but the biggest proponent of running models locally, and I do so myself a great deal. But one also has to be realistic about the capabilities of what you can do in a 16GB GPU these days. I already said an extra small Q2 quantization was effectively lobotomized so I didn't want to repeat myself.

This person has basically run into the limit of state of the art for even a modestly sized local model (this isn't deepseek v4 flash 0731 Q8 which I am running myself locally on a great deal more hardware), this is a 27B dense, but they're just not going to have a good time if they expect good quality results out of a Q2. The choices are either upgrade hardware or pay for external inference.

kennywinker 2 hours ago||

Fine, but they already said they are using a Q3, so Q2 being unusable (disagree, but whatever) isn’t helpful new info.

jwr 1 hour ago||

These are very good!

I'm hoping for speed improvements because the only problem running the 27B model on my Macbook pro (M4 Max) is the speed: 20 tokens per second. I benchmarked and MTP actually makes things slower, so I disabled MTP altogether. I'm hoping there will be some breakthroughs or optimizations that will allow me to run this at 30-50 tokens per second, which would make a big difference.

m1keil 1 hour ago|

I have a 36gb M3 Max. I tested it across quite a few different options: llama.cpp, oLMX, ollama with different options.

So far ollama managed to be the most performant of them all. I will get 30 to 40 tokes/sec with it when using the -mlx version of Qwen3.8.

Whatever the sauce the ollama folks baked into the mlx + MTP mix is currently working the best out of the box.

johndough 14 hours ago||

Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time.

I could of course download and test myself, but that would take days with my internet connection.

InvertedRhodium 13 hours ago||

I tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer.

I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything.

After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate.

Q4_K_P

Balinares 13 hours ago||

I anecdotally observed the same.

Interestingly, it also seems to tend toward self-correcting, which makes lower quantizations borderline usable. There'll be more faffing around, but still converging toward a solution. I wonder if that's a deliberate product of its RL.

danielhanchen 12 hours ago|||

We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc)

We do plan to do larger benchmark suites though!

johndough 4 hours ago||

Great to hear that you are planning larger benchmarks! I am particularly interested in longer-running tasks with many steps and self-correction. Divergence is fine as long as the model can still solve the task, which Divergence-300 @32 does not measure.

The current benchmark suites that frontier AI labs use are probably a good fit, e.g.

https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20com...

https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20...

https://www.anthropic.com/news/claude-opus-5

https://openai.com/index/gpt-5-6/

But guessing from your current benchmarks, I assume that you are severely compute-constrained. What is your time budget?

zdragnar 11 hours ago||

Purely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait...

It takes forever, but it does actually get around to making things work, and it is more thorough and produces better code than previous qwen models. You just need to let it run quite awhile.

zenoprax 9 hours ago||

I've seen the same thing. I tried the "superpowers" meta-harness and gave it a simple web app task and it spent 4 hours to make a basic timer app. I might try restricting the amount of thinking it is allowed to do to 500-1000 tokens.

stkdump 8 hours ago||

There is a native reasoning effort setting. It defaults to xhigh, I guess to get the best benchmark results, but you can just run it on medium or low instead, or for simple things even disable thinking outright.

lifepillar 5 hours ago||

According to this guy [0], medium is the level that tends to produce way less tokens in agentic workflows ("low" may output less per response, but then the model makes more mistakes, so it needs to iterate more).

[0] https://m.youtube.com/watch?v=z64J6bC16iQ

throwa356262 15 hours ago||

   "We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller"

This is crazy! But has anyone tried these lower quants on real projects?

Aurornis 12 hours ago||

I tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly.

If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.

andai 6 hours ago|||

I wonder if this would help, or if it solves different kinds of errors.

Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks

https://news.ycombinator.com/item?id=48192383

Havoc 11 hours ago|||

What setup are you using to do said private evaluation? Software wise I mean

kennywinker 15 hours ago||

Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts

Systemerror7A69 12 hours ago||

Since it seems like this not only improved sizes but also performance I can't wait for some benchmarks and comparisons. If you don't have a separate GPU for inference, every single GB matters so a comparison between specific Q4 Quants is really interesting to me.

Currently I very much can't decide between going for a bit of a lower Q4 Quant to squeeze out a bit of buffer and ctx or wondering if a slightly higher (IQ4_XS vs Q4_K_M/XL) is worth it

jjcm 11 hours ago||

No Dynamic 3.0 NVFP4 quants just yet from the look of it, as a heads up. Would love to see how those perform relative to others on the curve.

mike-the-brain 15 hours ago||

Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?

smcleod 14 hours ago||

Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.

kristjansson 12 hours ago|||

Just quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at higher accuracy, checking on representative data, and repeating.

[0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...

verdverm 12 hours ago||

Quantization is typically very cheap and fast. It can even be done on hardware that does not fit the model, by processing the weights layer by layer.

I use this project: https://github.com/vllm-project/llm-compressor

ankushdograuk 1 hour ago|

Waiting for MLX version

More comments...