Posted by neomindryan 1 day ago
I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat
This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact.
even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.
I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks.
Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6.
I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.
There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences.
You are better off with a 8b FP4+ model then a 27b Q2 model.
But do you need to run every small problem through a 10B-30B model?
We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code.
Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.
Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin.
I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.
Actually probably yes: text analysis (magazine articles) by LLMs in the ~30b .. ~120b range failed miserably (and also randomly - the rare cases of proper interpretation occurred among the failure cases) with the main public models of around one year ago, tried extensively.
So, yes, you can employ an ~80IQ only if you will expect the related quality.
I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.
Thanks for being skeptical, I maintain a llama.cpp-based client and it’s frustrating how high expectations are for local AI bc the median effort level means people mostly assemble their expectations and understanding via marketing soundbites
Is that likely, do you think?
There is a proposal in the USA to restrict LLM access. This will only have us depend more and more on open source models and their providers. And cause a drain of research in those areas in which it will be impeded.
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text
You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
Working with models that are super fast, but slightly dumber (like mimo-v2.5-pro-ultraspeed) is amazing, I feel like I'm still the one that's actually making every decision.
With a fast enough model, I can iterate on the UI of a given screen 4-5 times before Opus finishes its first attempt.
Keep your mental context in your brain is critical
Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.
Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway.
But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.
The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It's called a "programming language".
The mistakes are always in the transfer from human to machine. I still find a high-level programming language to be the best way to express my intent. Humans will make mistakes in the hand-off to AI just like they make mistakes in the hand-off to code, but at least code is deterministic.
This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single fucking control. For example, I am completely immune to `rm -rf` style fuckups because I wrote rm-safe years ago and it is mapped to `rm` in any environment that the LLM will run in: https://github.com/pmarreck/rm_safe
In any event, it's not a replacer, it's an augmenter. Nothing will replace humans, because we are the stakeholders; it may shift them around, though.
> The correctness of an application is limited by your ability to understand and describe what you need.
If you write code in Lean 4 or Idris 2, you may not completely understand why it is or isn't correct, but their respective compilers will certainly prove it to you one way or the other.
We already are perfectly functional with incomplete understandings. An entire generation of web developers have had successful careers without having a single clue how compilers work or how machine code works. Now is there a type of problem that sometimes comes along that DOES require deep understanding? Of course, that's when you call in the heavy artillery. Or the advanced LLM. Or both. See my point?
No, the prover can only prove that the implementation matches the formal language specification. That's different from the application being correct.
Likewise for image assets for CMS and ecommerce sites, now instead of hiring agencies, many of those images are AI generated.
Just because it hasn't come yet to some coder jobs, doesn't mean others haven't been already displaced by it.
At 5 tokens per second and unknown prompt processing speed, you may need a very extra long lunch break depending on your codebase.
These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.
Let the machine do its work while I relax, I’ll check up on it later.
If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low.
It’s not going to be cheaper than paying API prices for the model though.
This is how I used to think about my 3D printer, but FWIW the way my actual thinking and planning works, print speed really matters. Not for the final print, but for iterative work and test parts, it is obvious that either having a fast printer helps. Having multiple slow printers also helps, but there are only so many areas of a design you can iterate on at once.
At the moment my own LLM use is experimental and iterative, and I definitely favour the faster MoE models for much of what I am doing, even if I might in principle prefer to get the final work done in the slower ones.
It is common for agents to just stop because overload or some API error hijinks.
Or you get a TUI question that is blocking.
In general you’re right though, staring at tokens from agentic is not time well spent.
Some of these I’ve built custom harness around in iterm2 though.
That Xeon is going to be devouring power.
I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.
Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
It says so right in the readme. They’re not hiding anything.
Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
Like, obviously faster is better and the entire point of computers is to do things faster, but I find it kind of surprising how many people consider 9 tokens per second—heck, even much less than that—to be unusably slow. It's still automating a class of task that virtually no one before 2022 was able to automate, and it's faster than basically any human can write code.
I guess the better question is, why would you use a 9 token per second system when you could use a much faster cloud model. Obviously, if you want speed, that's the way to go. But a lot of people seem to find the idea of sending their requests to a third party server untenable. If 9 tokens per second is the best you can do—I don't know, that seems usable and useful to me.
The biggest argument in favor of running local models doesn't seem to be privacy at all, it's the fact that you can't run out of tokens even if it is a bit slow.
its pretty good for building any kind of research report using search engines and it manages context corrextly.
This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
On the contrary, that's ultimately an excellent baseline capability for most casual LLM users. Especially if you add the ability to fire off multiple requests over time and work on them concurrently (at least for not-very-long contexts), which is quite natural if one can implement some sort of continuous batching.
Even quantizing the model (with the ensuing loss in quality) is not an absolute requirement, quite unlike e.g. ensuring that a local model can run fully from VRAM on a typical consumer dGPU.
In general, ISTM that people overemphasize real-time or near real-time response which is a rather terrible fit if your goal is efficient LLM inference of near-SOTA models on typical local hardware.
What are the LLM standards?
Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.
Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.
It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.
If DDR6 comes out that will actually double the effective memory bandwidth on next generation computers. Most computers will reach a good fraction of a Strix Halo system's memory bandwidth and the next generation of Strix Halo will reach Macbook Pro levels of memory bandwidth. Of course there is no guarantee this will happen by 2027 but DDR6 will probably exist in some form by the end of 2027.
I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available
I also replaced my crap implementation of the oai server following yours, I hope it's all right. I did add a mention to your project and this fact in the README. It is a lot more barebones than what you have, but I have to admit that it works really nicely for me
I am getting [ Prompt: 91.3 t/s | Generation: 171.8 t/s ]
This is on a GPU (RTX 4060)
Is this decent?
I am curious about the decision to not use GPU since this is for Apple Silicon.
Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?
Second off, this doesn’t work from a power consumption standpoint. When I run qwen3.6-35b, a far smaller model than op is suggesting, power usage spikes to 150-200W during inference. To fit a 1T model in the palm of my hand, the amount of processing required doesn’t fit the amount of power available.
Now I’m not saying this will never happen - there are some great leads, e.g. burning models directly on to a chip - but op’s scenario is definitely not happening in two years. Maybe 5, a lot more likely 10, unless of course local ai is made illegal
I absolutely agree that models are going to advance on to “edge” hardware over the next few years by becoming small + specialized.
* Software inference optimizations
* Heavy quantization
* Chips with hardcoded transformer architecture
* Much cheaper HBM
* Much sparser models - 1T total with ~1-10B active params e.g.
* Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.
Absolutely. I'd be surprised if they couldn't 2x performance in the next year. Still doesn't make a 1T model fit on your phone.
> * Heavy quantization
I think this is a dead end if you're trying to fit a 1T model into a phone. Makes much more sense to train a model that's designed to be small, than train a model that's smart and then quantize it into stupidity.
> * Chips with hardcoded transformer architecture
Totally, this will probably work great. Now good luck booking fab time any time in the next 2 years.
> * Much cheaper HBM
Totally, this will probably work great. Now good luck booking fab time any time in the next two years.
> * Much sparser models - 1T total with ~1-10B active params e.g.
Fewer active params helps with the speed of token generation, but if the whole model doesn't fit into ram it doesn't solve the issue of having to constantly stream portions of the model from disk to ram.
> * Not to mention - 2 years of today's frontier models writing RTL and kernels at superhuman levels.
IMO this is a delusional myth-making idea being sold to us by ai companies. Machines that generate output based on statistical averages won't generate genuinely new ideas. They can help us try out ideas faster, but they're simply not capable of the kind of creativity and understanding required to push a field forward, except incrementally.
> Datacenters are willing to pay $50k for a single high end GPU.
That doesn’t change until production capacity exceeds the datacenter demand. When that happens, they’ll start selling them down the market until it eventually reaches phones and toasters and whatever. But not in two years.
That is no where near decent at all.
It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)
Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
I know! That's my point! You're a poor salesman of a coding environment/tool if you don't even use it yourself for coding...
that would have run into a race condition unfortunately ;)
but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
Not really, you start small, bootstrap as soon as you can, and off you go. Requires a good model though ;)
If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.
I estimate that the server consumes probably around 500W during inference.
In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.
But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.
The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.
EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.
In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices.
For sonnet the equivalent token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The cloud is 10x more expensive for generation and close to 50 times more expensive for processing.
My guess is that this math gets less good with MoE (because you will be limited by VRAM, but clouds won't).
Try dropping the power cap on your GPU if it supports it; you can often get much lower energy usage with minimal loss of tok/s (particularly during generation) than whatever the GPU defaults to. There's a sweet spot around 200W on the GPU I'm currently testing that gives me about ~75% of the max pp and 97% of tg while using 100W less than the default/max 300W power cap -- and the card runs much quieter as a result.
It will be feasible for everyone to have 20 different agents running at all times. A new world is coming
Within a short time I think open source models will all be getting good and efficient enough to make it viable to serve this on 2nd hand hardware for cheap. All it will take is a nerd in every small community to pool together a few hundred bucks initial outlay, and then ongoing costs are near free without electricity to pay for.
The commodification of the hardware needed is probably a larger factor, because by the time a baseline computer has enough RAM and processing power to run a desired LLM, that hardware will be efficient enough that the extra electricity usage is nominal.
Maybe, but for how long? Prices keep going up, and every new model eats more and more tokens...
The privacy nuts have a better leg to stand on, but even then it's hard to believe that they're using on-prem AI to replace SOTA model inference. As cool as local LLMs are, a lot of the stuff people run is a novelty.
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.
I was responding to a lot of the comments saying this was a reasonable way to avoid paying for tokens or subscriptions. I don’t want anyone getting the wrong idea that this is a way to save money if that’s their priority.
What does it make clear? That I can replace the space heater my wife runs 9 out of 12 months of the year with a home server? And effectively get $0.00 per token during those times?
In houses running A/C year round, sure there'd be some impact, but in all the places running heat, doesn't seem that it'd move the needle on power bills.
There are startups whose entire business model is "cloud server as a home space heater" (aka "data furnace") ...
That it’s pulling a lot of watts.
Good for you if it’s replacing electric space heaters.
https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
I'm excited to get my mitts on it on Friday when it finally arrives.
Here's some of the resources I came across if you're interested in reading.
https://echalupa.com/blog/mac-pro-6-1-llama-cpp-firepro-d300...
https://matthewgribben.com/blog/mac-pro-6-1-llama-cpp-firepr...