Posted by gitpusher42 8 hours ago
I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal.
I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory.
The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are included.
The trick is to keep the shared part of the model and the KV cache in RAM, then stream only the routed experts needed for each token from SSD. An SSD is way slower than RAM, so the runtime uses a small expert cache and bounded parallel `pread`. While those reads are in flight, the GPU runs the shared part of the layer.
I ran more than 100 experiments. Most didn’t work. A few got me here. The experiments are described in the GitHub repo.
It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
I also added an experimental OpenAI-compatible local server. It supports streaming and tool calls, and reuses one prompt prefix from the KV cache.
Try it! The Mac app is easy to install. On the first run, it will download 15 GB of weights from Hugging Face. The model is surprisingly capable.
I would love any kind of feedback!
Windows PCs would require a completely different approach
Did LLMs arise because a) humanity created circuits so large and so fast and so easy to use in parallel that only then did it become possible to run an LLM, or b) because sufficient data useful for training was accumulated such that experiments in different neural network arrangements could be done to see what came out?
My hunch is (b) and so I further wonder how far back in time could we have made a usable LLM if we had only known to try? E.g. can you run any sort of LLM on a VAX 11/780?
I saw a pretty cool project to run an llm on an esp32 device https://github.com/slvDev/esp32-ai
I think we strongly need something like that (shameless plug, I tried to build something around bitNet for the same reason: https://github.com/nickyreinert/bitNetRTR).
But at the end, all aproaches I saw, however genius they are: the actual results are always a mess. It's a better chat buddy, nothing else. It's e.g. far away from an decent coding assistants. I fine tuned Gemma with domain specific knowledge. Running it on a 16GB VRM GForce. Even then it's okai'sh but far way from a mind blowing experience. I ran some of the promised open source model on my 36GB MBPro M3, in Pi, Hermes, Continue. Can't compare the results to what Claude or Codex are offering.
You need at least something that's far away from consumer hardware, like those 7k'ish GForce machines with 96GB VRAM to get an idea of a good competitive model.
But... please, proof me wrong! =)
You are not really accessing the biggest frontier model every time, and you're not really doing an end-to-end LLM request on each prompt.
I would go so far to say frontier models have peaked and improvements from here come from clever (or very elaborate) harnessing. "LLLMHs" - Large Large Language Model Harnessing !
Do I understand correctly that Ollama doesnt do that, and that’s why responses hang forever on a M3 running the same model through Ollama?
afaik ollama relies on llama.cpp and mmap. mmap loads pages on demand and doesn't use the same explicit cache or parallel reads like my engine. Most likely ollama/llama.cpp will be way slower in this case
what are the limitations