Posted by thm 17 hours ago
I always thought Apple was in a good position because unlike their peers they didn’t burn billions of dollars trying to build an AI model that has no financial moat around it.
I still think they’re in a good position in comparison to their tech peers and we will know even more when some of the new computers get into the hands of some of the tech reviewers.
I believe the new computer’s will be pretty good hardware wise what I’m interested in, is the Apple software support for connecting several Mac computers together, and some of the other (new?) software that Apple may have written in house to support those who want to run AI software locally that is just as important as the new hardware.
It's old Steve Jobs logic. Works backwards from the customer experience to the technology (they're the only big player I see doing this).
I would wait till the ram crisis is over to fetch a future 64gb ram gpu to run Q8 models. Cloud inference until than.
A year ago you could get an M4 Mac mini for $399 on sale and now the same one used goes for over $700. The general AI RAM/SSD spike is part of that but there was also a huge demand spike for small, powerful, desktop machines that could be easily configured with these workflow tools.
It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
64GB+ or dedicated 48GB (2x24 on GPUs) is IMHO absolute minimum.
- 16GB for the weights at Q4
- 9GB for the full 256K context at Q8
- 7GB spare for overhead and system.
The problem is that these Macs have 32GB of slow unified memory.
Edit: I'm thinking of a headless Mac mini, if you meant running it on the same machine you're using of course you'll need more memory, but LLMs are best served from a headless server so that's what I'd recommend.
32gb of unified memory is enough enough for system to be used for anything other than LLM generation.
What? LLMs are best served from a massive PD disaggregated cluster of B300s connected via NVLink.
If you're running LLMs on a Mac Mini, it's because you want to run local, not because it's the best setup.
https://www.canirun.ai (five months ago: https://news.ycombinator.com/item?id=47363754 377 comments)
32GB is not enough RAM. I don't even own a device with less than 36GB at this point, and that device I only have because my employer is being cheap. 64GB is a reasonable starting point for running local LLMs + normal tasks. 128GB let's you really run most smaller models like Qwen 27B and 35BA3B with good context. Even Qwen3.8-Flash-Next runs in 128GB with a 4-bit quant.
32GB would be limited to running models like Gemma4 12B and smaller dense Qwen versions like 9B unless you were using very small quants which damages quality of response.
Is this the best? No. That's why I said the sweet spot. Getting from 16GB macs to 32GB is perhaps possible. Jumping to 64GB or 128GB as the default is simply unreasonable right now.
What sorts of things are you doing with the local LLM? Anything interactive? Should I take another look?
And I don't remember to have been able to have pushed to 200k context Qwen 3.6. 3.8 is running on my RTX 5090.
EDIT to add that you need to reserve 8GB for the system if you don't want to cause problems on macOS, which means 32GB RAM = 24GB max for model + context. It takes 18-19GB to load a 4-bit quant of Qwen3.8-27B, so I'd be really surprised if you can actually get a 200k context window. You need to fit within a 24GB WSS (which is generally a more constrained RSS) to get stable performance on 32GB RAM.
I have no idea why. They could be so successful if they leaned into local AI.
I also don't think they do "technology". For example, containers have been around for a long time, and apple didn't show up. (I know they have some support now). Imagine an apple-native docker/podman doing something like FROM macos:10.12
I was actually surprised when they did their own chips. I figure it was about control.
But apple have to own everything they do so they’re in a bind