Posted by gitpusher42 7 hours ago
I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal.
I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory.
The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are included.
The trick is to keep the shared part of the model and the KV cache in RAM, then stream only the routed experts needed for each token from SSD. An SSD is way slower than RAM, so the runtime uses a small expert cache and bounded parallel `pread`. While those reads are in flight, the GPU runs the shared part of the layer.
I ran more than 100 experiments. Most didn’t work. A few got me here. The experiments are described in the GitHub repo.
It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
I also added an experimental OpenAI-compatible local server. It supports streaming and tool calls, and reuses one prompt prefix from the KV cache.
Try it! The Mac app is easy to install. On the first run, it will download 15 GB of weights from Hugging Face. The model is surprisingly capable.
I would love any kind of feedback!
The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly
What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.
I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.
The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.
If you need maximum speed and flexibility just use MLX
Anyone got recommendation about what local model to use for what purpose ? I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, granted each one is specialized enough.
[1] : https://github.com/JustVugg/colibri [2] : https://fireworks.ai/blog/kimik3-fable
I am using Gemma for a few tasks simply because it’s “good enough”.
Otherwise IMO it codes about as well as the Qwen MoE for PHP and SQL. It's a fully impressive model (though it is not as mindbendingly impressive as the 12B, which is outrageously good for its footprint)