Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

Posted by gitpusher42 6 hours ago

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac(github.com)

Hi HN,

I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal.

I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory.

The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are included.

The trick is to keep the shared part of the model and the KV cache in RAM, then stream only the routed experts needed for each token from SSD. An SSD is way slower than RAM, so the runtime uses a small expert cache and bounded parallel `pread`. While those reads are in flight, the GPU runs the shared part of the layer.

I ran more than 100 experiments. Most didn’t work. A few got me here. The experiments are described in the GitHub repo.

It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.

I also added an experimental OpenAI-compatible local server. It supports streaming and tool calls, and reuses one prompt prefix from the KV cache.

Try it! The Mac app is easy to install. On the first run, it will download 15 GB of weights from Hugging Face. The model is surprisingly capable.

I would love any kind of feedback!

506 points | 177 commentspage 2

tpurves 4 hours ago|

Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two?

my impression right now is that M5 gen is on the cusp of practicality for local inference.

If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainstream.

Joeri 2 hours ago||

Without fundamental model architecture improvements the practicality largely depends on how Apple increases memory bandwidth.

Memory bandwidths (* = rumored):

  M1:       68 GB/s
  M2:       100 GB/s
  M2 pro:   200 GB/s
  M2 max:   400 GB/s
  M2 ultra: 800 GB/s
  M5:       153 GB/s
  M5 pro:   307 GB/s
  M5 max:   460 GB/s
  M6:       200 GB/s*
  M7:       240 GB/s*
  
  Nvidia 4090 1008 GB/s
  Nvidia H100 3.35 TB/s

Basically what we're looking at by the M7 generation is a tier shift, where the base M7 can do what the M2 pro did, and every tier moves up accordingly, with the M7 ultra becoming competitive with nvidia dedicated consumer hardware.

hatthew 4 hours ago|||

My assumption is that the difference is 90% from more memory. I'm making several assumptions because nothing here looks groundbreaking so I don't care to dig deeper, but the model + KV cache definitely cannot fit in memory on the 8GB machine, but probably can on the 24GB machine—or can at least get close. Assuming that this benchmark makes use of that, skipping SSD streaming will speed things up massively (I would have guessed much higher than the reported 6x speedup).

Aurornis 4 hours ago|||

I have an M5 128GB. Being on the cusp of practical is a good description. It will run, but prefill and token gen are still slow relative to my consumer GPU box.

It also gets very hot. If you’ve never heard the fans on Apple Silicon really spin up, it could surprise you. Makes the full GPU setup feel quiet by comparison.

I think after the hardware market calms down the ticket is going to be a light laptop with a second dedicated inference server on the network.

anthonypasq 3 hours ago||

i wonder if Apple will eventually ship proprietary models with burned into the silicon for all local workloads

https://eu.36kr.com/en/p/3904844399445638

trollbridge 4 hours ago||

The speed of this perfectly correlates with the memory bandwidth of an M2 vs M5 Pro.

docheinestages 4 hours ago||

Is there a pipeline or approach to do this to any model? I'm particularly interested in Qwen 3.6 27B as it's the best for its size at the moment.

gitpusher42 2 hours ago|

This approach will only work for MoE models. There is a Qwen 35b-a3b. You just need to do GPU stop after router and read the requested experts to ram. And it is possible to build similar engine for this model (or feel free to adopt my engine) Not sure about generic approach for now, but coding with ai agents is relatively cheap now, you can try it

luciana1u 4 hours ago||

26B in 2GB is the engineering equivalent of fitting your entire apartment into a storage unit and still having room to pace

gitpusher42 3 hours ago|

Thank you! Thankfully this is purely software engineering problem. And as usual there is no free lunch. You trade speed for lower memory usage

hsienchuc 5 hours ago||

I've run local video generation models on an 8GB graphics card and know firsthand that nothing runs smoothly when memory is insufficient. So seeing 14GB of weights crammed into 2GB of RAM is impressive.

If running continuously for over an hour (like an overnight batch task), will a fanless MacBook Air overheat and throttle? Can the SSD handle the continuous weight reads and sustained output speeds?

Great work, congratulations on the release!

gitpusher42 1 hour ago||

Thank you very much!

I think it will throttle quite soon, but I haven't tried runs longer than 30minutes with this engine.

However, there is no constant load on ssd or gpu. i/o and gpu work are alternating and there is a brief idle periods for each i/o and gpu during inference (because gpu waits for i/o and after that i/o waits for gpu)

m00x 5 hours ago||

This is where MoEs shine though. You don't need all experts in memory at once. Diffusion inference doesn't have sparse inference.

agcat 1 hour ago||

This looks great. going to try it

gitpusher42 54 minutes ago|

Thank you! Let me know how it goes and share your tok/s results

hatsunearu 2 hours ago||

What's the best option if I have sufficient RAM to store the 14GB model?

gitpusher42 2 hours ago|

For Mac I would start from MLX engine. For exact model choice it is better to check bench results, and select model based on your need. A lot of good feedback about Qwen3.6, but I haven't used it in my tasks

freediddy 3 hours ago||

I keep seeing more and more LLM models being loaded by incredibly under-powered machines. Is the GPU/memory crisis all lies? I get that running on an RTX 5090 will be much faster, but if we can use main memory instead of VRAM and get barely usable results, what is going on?

efficax 3 hours ago||

all the effort in this space is going into multiuser, datacenter workflows for high throughput inference. running in resource constrained environments is not where the money is. but it will be, especially if we look forward to a world where having 512gb unified ram is normal for "workstation" machines. The semiconductor space is slow enough to respond that it's likely it will take a few years before production capacity has ramped up enough to get us past the current supply crunch but it seems inevitable to me that we'll be able to run huge models like kimi 3 locally in the next few years (maybe 2029/2030 for it to really be affordable)

gitpusher42 2 hours ago|||

I think it depends on usage pattern. You trade speed for lower memory usage. Maybe engine specialisation and faster SSDs is the future for local inference, who knows

MaxMatti 3 hours ago||

Or maybe it's just lazy programmers, wouldn't be the first time.

tracker1 4 hours ago||

This is actually very similar to some ideas I've been having for a while... that having a smaller entry model that knows enough about "expert" models that themselves are smaller to hand work over to could be better/faster/lighter in terms of working through real problems vs the megalith ones we currently use. Highly distilled experts and coordination with a fallback mode to a larger model option.

mxmlnkn 5 hours ago|

This sounds really cool. My intuition was that the selected experts might change heavily for each token, resulting in slow SSD loads for each token. This seems to be wrong. Did you create some statistics on how often the experts need to be changed? What is the longest token run without any expert change? What does such a token run look like? In which cases do experts change frequently?

gitpusher42 5 hours ago|

The full route changes almost every token. The cache works through partial reuse, about 40% of experts repeat on the next token and 57% within two tokens, cutting I/O from 166 to 88 ms/token on M2 Mac.

The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly

More comments...