Top
Best
New

Posted by datascientist 5 hours ago

Pete warden on local AI, open models, voice interfaces, the future of hardware(thedataexchange.media)
173 points | 157 comments
lrvick 3 hours ago|
I run 8x r9700s at home which lets me run up to 2x 1M context sessions of GLM 5.3 flash at ~50tps or 12x sessions of qwen 3.8 flash next at ~100tps.

I exclusively use this setup for all AI usage from fixing obscure reproducibility bugs in software that once took me weeks, to compiler bugs, to kernel hardening and hardware drivers, to building a complete unprivileged and security hardened replacement for systemd in rust. Most of that was done using Qwen 27b back when I had only 2x r9700s.

There is literally not a single problem I have faced in 25 years of systems engineering that I cannot solve now 10x+ faster using my local LLM setup.

I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.

lrvick 1 hour ago||
For those asking in threads. 4x r9700s will cost about 6-8k depending on how patient you are with ebay auctions.

From there any motherboard and CPU that can give you 4x slots of 16x Gen 5 PCIe slots positioned such that you can install 4x cards is probably fine. Ideally with a CPU capable of memory encryption.

My latest pairing is a ASUS Pro WS W890E-SAGE SE For $800 open box, and a Xeon 654 for $1300. My first build was an AMD EPYC rig for similar cost a few months ago using a Gigabyte ME03-PE0 and a AMD EPYC Siena 8224P for about 2k combined. I specifically wanted one amd and one intel machine for reproducible build diversity which most probably do not care about.

Do not get too attached to specific cpus and boards. Get what is available for cheap with the needed pcie bandwidth open box on ebay from reputable sellers to save thousands of dollars.

Be sure to get a power supply that can handle all 4 cards. I run Silverstone Hela 2500s at about $800. Do not go cheap on power supplies. It will end in pain. Can confirm.

Then 2-3k for 64gb in DDR5 RDIMMs. This is the most painful thing to purchase but it is what it is.

Not only do I recommend this setup, I plan to build more of them to share with my team.

If you are patient and buy used, a 10k budget can get you a very powerful local AI rig running 100% FOSS software/drivers even today.

back_to_basics 2 hours ago|||
>I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.

Which is exactly why said power and utility will soon be BANNED under a myriad of false aegis'.

An economy that almost exclusively extracts ECONOMIC RENT from the working class, won't permit any "leakage" or autonomy.

agcat 2 hours ago||
Did you turn down this for internal use-case? OR for your core product? I am keen to learn how are people running stuff locally for their core product to be able to make it easy for their customers to use.
lrvick 2 hours ago||
I have no interest in having my business or my personal AI usage being reliant on any single third party. We currently have no customer facing LLM products, but if we did they would be only options we can run confidentially which no providers can do ATM to our standards. Current plans are just racking GPUs like I did at home.

Most of my peers do use OpenAI and Anthropic for development work, which pains me, but I will never tell someone else how to do their job if the end result meets my quality bar.

I lead core systems engineering building custom enclaves and operating systems and I do 100% of my work with hardware I own on my own property. When the internet goes down, I do not care and keep building.

0x445442 1 hour ago||
If you don't mind me asking, how do you drive the models? Right now it seems like everyone is feeling their way around in the dark about how best to steer the work. AWS AI-DLC, SpecKit, BMAD-METHOD, TwitTech Influencer Pile Of MD files...

It all seems hopelessly adhoc at the moment.

lrvick 1 hour ago||
My own fork of jcode stripped down to only my needs running in herdr. That is it. Just lean rust code and no javascript nonsense needed.
MomsAVoxell 2 hours ago|||
I exclusively only use local models. I do not subscribe to anything, and I have my agents wrapped and inspected.

2xDGX Spark cluster is my current load, it is sufficient for a great deal many wonderful, wonderful things. It isn't fast as consumer AI, but I have nevertheless been able to do wonderful things with it for the past year'ish.. I'd get a couple more, but indeed they've tripled in price, so yeah .. things will just do for now, and it's fine. Pretty great, actually.

I think the way we control AI is we air gap it. My cluster is air-gapped.

txdv 3 hours ago|||
whats electricity bill and do you not fear that your hardware will be obsolete while others will have access to better models on faster hardware?

i just calculated that it should be Per month (30 days) 380euro in my region, which is a lot after an initial 20k.

I have to say, no limits stopping sounds nice though.

lrvick 2 hours ago||
> whats electricity bill

$300/mo in the bay area for 1kwh running 24/7, at least until I get solar installed, then free.

> do you not fear that your hardware will be obsolete

If it never gets better than it is right now, I cannot think of any problems in software I cannot solve locally, so I am all set for a while.

Though also, the capabilities on my -existing- cards have gone UP dramatically in the past 6 months since as the local LLM community keeps optimizing.

Sure in 3 years all these cards will be a lot cheaper than I paid, but having them for the next 3 years is absolutely worth it for me.

This costs me less than a used car. Most people chase car and phone upgrades every 2 years. I tend to get something good enough and upgrade once a decade. Depends on your goals really.

paul7986 1 hour ago||
Are you able to interface with it entirely through voice?
lrvick 1 hour ago|||
I could easily, but not been a priority yet.
derac 1 hour ago|||
why not? that's very easy to do with tts.
paul7986 1 hour ago||
pardon what is tts as im just beginning my journey into building my home local/private AI companion.
thunderbong 33 minutes ago||
text-to-speech
wez470 3 hours ago|||
How cost effective had it been? I don't want to rely on 3rd parties either, but my experience is it's super expensive to run locally versus using a provider (both in terms of hardware costs and ongoing electricity usage).
lrvick 2 hours ago|||
No third party can ever provide the privacy and sovereignty of being able to run locally with abliterated models. I would do it this way even if the providers were much cheaper as I value those properties a -lot-.

In practice though, I can produce about 3B tokens/month of Qwen 3.8 Flash Next which is the model I crank almost 24/7. That would cost something like 1k/mo on OpenRouter. My setup costs about $300/mo in electricity, though getting solar soon to drop that to zero.

cortesoft 2 hours ago||
It doesn’t really drop the cost of electricity to zero; you have to amortize the cost of the panels over their lifetime, plus the opportunity cost of the capital outlay (or interest on the loan)
lrvick 2 hours ago||
That is fair. Still, dramatically reduced cost and reliance on one less third party.
ahartmetz 1 hour ago|||
Running locally is not a problem if you allow it to be slow. I get about 3.5 tps with Qwen3.8-27B:UD-Q4_K_XL on my 64 GB laptop and about the same tps with Qwen3.8-Flash-Next:UD-IQ4_XS (IIRC, or UD-Q4_K_XL) on my 128 GB desktop with a 16 GB GPU.
random3 3 hours ago|||
Can you share more, or direct us to more resources around local setups? For starters what's the upfront cost, what are the space, weight, energy requirements, etc.
lrvick 2 hours ago|||
Really depends on what GPUs you go with but I hate proprietary software so I went with AMD AI PRO r9700 cards which you can still pick up used on ebay for $1500 used on auctions. Started with 1 then expanded over time as I won auctions. Though on a lower budget my $400 16gb 6800 XT can run Qwen 3.8 27b which I could absolutely get by with at significantly reduced speed.

My entire 12u rack has roughly the same capability as an H200, though at maybe 15k all in with mostly used parts vs 50k for a used H200 checking todays ebay prices and that is -just- the card you still need to build a system for.

Consumer cards are a lot more power hungry than enterprise cards and they take up more space. I run my setup next to my dryer, running on 240v for 15% better power efficiency.

Also using transparent memory encryption on a Xeon and an EPYC for confidential and private inference guarantees no third party can offer me. Of course I can run abliterated models that do any security research I want without refusals.

Strata and Radiance are the state of the art ATM for running big MoE models on small consumer GPUs at high performance.

Leif24 2 hours ago||
Re: running qwen 3.8 27b on a 16gb card - that isn't a MoE model, was it just heavily quantized? How much context window were you getting?

Just starting to dive in on local AI with my own 16gb card, and the struggling with evaluating what tradeoffs are worth it.

lrvick 58 minutes ago||
On a 16gb card check out Bonsai2. I run it on my 6800XT.
vablings 2 hours ago|||
There are tons of resources out there on how to run a local setup. Once you have something that's somewhat functional you can then have the system self optimize your software stack to suit your needs.

As for cost a 9700 is running at about 2k USD MSRP right now and the rated power is 300w.

dghlsakjg 2 hours ago||
How does this make you less reliant on third parties?

It seems like it just shifts which third parties you rely on. I think local inference is cool and your setup sounds dope, but you still have third party hardware manufacturers, and third party models in the mix.

What is the advantage of this over OpenRouter, which lets me select for, jurisdiction, quantization, ZDR, etc.

note: My use case can't even come close to having this make sense financially, but I understand if yours does. Either way, that doesn't make the third party argument better.

runtime_terror 2 hours ago|||
Not the OP but I would assume: - no data retention policies to worry about in the first place - no monthly billing outside of power bill and capex - no jurisdiction issues - full control over your data - no network round trips or even worry about network outages in the first place
lrvick 2 hours ago||
Exactly. I own the GPUs and bought them all used on ebay. One time cost.

When the internet goes down, my productivity is not impacted, and I have total privacy on my work at all times. Also no rate limiting etc, and I can burn tokens freely.

bornfreddy 1 hour ago||
Maybe a naive question, but how do you even run 8x GPUs? Threadripper Pro? Anything special about OS / software stack?

Sounds interesting, though a bit pricey (if I'm reading the prices correctly it's 20k just for GPUs). Fingers crossed that the prices will fall.

lrvick 57 minutes ago||
Gave a full parts list as a reply to my original post
almondfestival 1 hour ago||||
it's the difference between a one-time transaction with a third party and an ongoing, perpetual third-party liability. you buy your inference hardware, you download your model, and if you wanted/needed to, you could run inference in perpetuity (or until your hardware broke down). compare that to anthropic/openai/etc being able to throttle intelligence, deprecate models, or raise prices on you whenever they felt like it.

sure, in a very general sense you're dealing with third parties in both scenarios, but if you actually own the hardware and model weights, you can control your own destiny for the lifespan of that hardware.

dghlsakjg 1 hour ago||
> it's the difference between a one-time transaction with a third party and an ongoing, perpetual third-party liability.

no ongoing liabilities except the power bill and opportunity cost of dropping $20k on hardware.

He isn't using OAI or Anthropic models. He is using models that area available for absurdly cheap from completely independent providers in multiple jurisdictions.

I get the base level argument, but from a practical point of view there are so many different dependencies that shifting one isn't really doing much.

drdexebtjl 2 hours ago|||
Is this some sort of gotcha? You didn’t build your computer from scratch with sand, sticks and pebbles, so you’re just as reliant on third parties?

If you own your own hardware, you don’t have to depend on and trust third parties continually, just once when you bought it.

lrvick 2 hours ago|||
This feels a bit pedantic, but my scope here is third party services, like Anthropic or OpenAI subscriptions.

If the power goes out, internet goes down, I fire up my generator and keep building.

We are talking about digital sovereignty and ownership.

I own the means of my production.

0xbadcafebee 1 hour ago||
Sounds like commie bunk to me! We can no longer sit back and allow Open Weight infiltration, Open Weight indoctrination, Open Weight subversion, and the international Open Weight conspiracy, to sap and impurify, all of our precious bodily fluids.
dghlsakjg 1 hour ago|||
> Is this some sort of gotcha? You didn’t build your computer from scratch with sand, sticks and pebbles, so you’re just as reliant on third parties?

I'm not the one claiming to be free of third party dependencies. Third parties built my hardware, they wrote the firmware in the hardware, they wrote the OS on the hardware, they designed the network stack that you might use to connect to a local server, they provide security updates to all of that software, they ensure that I have a stable power supply to run all this, etc.

I'm incredibly reliant on third parties, some of which happen to sell me tokens. I recognize that I live in a world where I will ALWAYS have a dependency on third parties, and rather enjoy the fact that I can use GLM flash on someone else's $20k computer for a few cents.

drdexebtjl 26 minutes ago||
Yes. They built the hardware. Now it’s mine, and they’re no longer involved, so I no longer depend on them. That’s how it works.
giancarlostoro 4 hours ago||
If Nvidia didnt abandon consumer GPUs we all could be running better models locally by now. The moat is companies like Nvidia not releasing better consumer hardware so they can partner with bigger companies so they can sell us overpriced hardware.

They release a new RTX series every two years, last time was 2022. (edit: I mistakenly quoted the year wrong, not sure why, but it seems obvious to most that Nvidia doesn't care about the local consumer market anymore)

Meanwhile major AI companies are investing in their own chips, if Nvidia wants to survive they really should not abandon consumers.

Hoping AMD steps up to the plate. Meanwhile the nextXbox and PlayStation consoles are going to have a rough time with their next consoles which will likely cost a thousand dollars or more is my gut feeling.

codehorses 4 hours ago||
Not defending Nvidia here, but the RTX 50 series was released in 2025, was it not?

Regardless, they aren't planning on sticking to the two year release cadence any more which does suck.

paxy 4 hours ago|||
RTX 5090 - release price $2000, average price today $6500

RTX 5080 - release price $1000, average price today $2300

Nvidia also stopped selling its own version of these cards directly to consumers and is happy with letting partners and resellers/scalpers control the price.

So it is pretty obvious that the company doesn't particularly care about this market segment.

giancarlostoro 3 hours ago|||
> So it is pretty obvious that the company doesn't particularly care about this market segment.

Which I think is foolish, every AI shop is making their own alternatives, it's not going to be a winning bet for them.

theaxeonthanksg 4 hours ago|||
[dead]
Buttons840 4 hours ago|||
Ironically, if consumer GPU progress stalls, then slowly PC becomes the best console. Everyone has the same top-end GPU that can run any game.
ghostbrainalpha 4 hours ago||
Or game streaming actually becomes a thing. Because the best new games only play in the cloud on computers we could never afford.

Almost like going back to the Arcade era except we all pay monthly subscriptions instead of 1 quarter at a time.

unrented7977 4 hours ago||
Game streaming isn't going to happen. It's been tried several times now and failed every time.
Buttons840 3 hours ago||
I guess gamers are tech savvy enough that they care about ownership on principle? Or why is it that game streaming hasn't taken off?
Lord-Jobo 3 hours ago||
Almost every gamer can immediately tell the difference in fidelity and latency. And only a very small handful of gamers have the internet quality for game streaming at low enough latency and high enough fidelity to be “sort of on par”.

The quality and internet connection stuff will improve over time. The latency problem essentially never will.

skinfaxi 1 hour ago||
I can see a world where the edge is built out a la Netflix. https://openconnect.netflix.com/en/
Tuna-Fish 3 hours ago|||
GPU-style platforms (lots of compute connected to a big shared pipe that connects to a large pool of fast memory) are inherently more efficient for centralized inference.

Local inference gets truly interesting when the proper next-gen, natively AI and not just a repurposed GPU, stuff starts to land. Initially things like compute-in-memory and fancy flash, and later possibly the truly exotic stuff like various RMM (read-mostly-ram, for example stacked FeFET on logic) tech.

IMO there is no chance for ubiquitous local AI to happen if that means people using GPUs. But the industry is chasing all possible solutions for more efficient inference platforms, and if something like pachinko machine architecture with FeFETs ends up winning, suddenly buying a local ai inference solution in a box is no longer as insane.

winwang 3 hours ago|||
If consumers were willing to pay more for GPUs like datacenters do, NVIDIA would have sold plenty of consumer GPUs. Or would you press a button to halve your revenue for some reason?

(I say this as someone who would have liked a 5090...)

lrvick 1 hour ago|||
> Hoping AMD steps up to the plate

Nvidia has nothing close to the Strix Point, Strix Halo, Gorgon Halo, etc.

My GPD pocket 4 handheld can run Qwen 3.8 27b only because of AMD making embedded mobile inference cheap.

bornfreddy 4 minutes ago||
> Nvidia has nothing close to the Strix Point, Strix Halo, Gorgon Halo, etc.

Is there a reason why DGX Spark doesn't qualify? (genuinely curious)

scronkfinkle 4 hours ago|||
> Hoping AMD steps up to the plate

AMD's aggressive marketing as an AI-first company leaves me pessimistic about that.

cyberax 1 hour ago|||
I'm running qwen3.8-flash-next at 4-bit quantization on two AMD Radeon AI PRO R9700 cards at 75 tok/s. And I can actually run two models in parallel at around 50 tok/s, so you can get by with only one card. I switched from NVidia to AMD for a fully open-source stack for my local inference.

AMD Radeon AI PRO R9700 is $1600 right now (I bought them last year for $1300).

They are also quite power-efficient. My setup typically uses around ~300W during inference.

Forgeties79 4 hours ago|||
No gut feeling necessary. They announced weeks ago that they are going to be $1200-$1800 (Xbox specifically, PS6 somewhere in that range too).
lubujackson 4 hours ago||
Not to mention the required monthly subscription... Income inequality like 1929, might as well bring sharecropping back.
jimbokun 2 hours ago|||
Yes, ten the home AI market would be similar to the serious PC gaming market used to be when they were the main market for the latest GPUs.
jjtheblunt 4 hours ago||
> we all could be running better models locally by now

Apple silicon says "hold my beer", which I know because i've been doing this on my 3 year old quite beefy mac studio, for example. it's good enough for many tasks.

vadansky 4 hours ago||
Can't wait for M7! I hope the rumours are true.
paxy 4 hours ago||
I don't see any difference between the conversations around self-hosting local AI and self-hosting regular software, which have been going on for decades.

Yes, there are a hundred valid reasons to self-host. But 99%+ of consumers aren't going to care.

People don't even care when the cost of hosting is ~free, like a simple home server. How many are ready to buy thousands of dollars worth of AI hardware, which could very well be obsolete a couple of years from now?

abdullahkhalids 1 hour ago||
One of my favorite stats is that Limewire used to be installed on 1/3rd of computers worldwide [1], i.e. people were self-hosting.

Why? Because Limewire was easy enough to install that even grandma could do it. There is not a single way to self-host even a simple html website without learning half a dozen software skills. People will self-host when it is click-and-play.

[1] https://marquettewire.org/3777966/tribune/marquee/the-life-a...

rockostrich 32 minutes ago||
> There is not a single way to self-host even a simple html website without learning half a dozen software skills.

1. Set up port forwarding on your router

2. Open a terminal

3. Run `python -m http.server 8000` in the directory with the html file

I'm not sure you could really count any of those 3 as software skills, but even if you count all 3 it's still half as many as you said it would take.

nvme0n1p1 7 minutes ago||
You stopped halfway. Is grandma behind a NAT? What do we teach her about next, domain registrars or DNS? What happens when she closes her laptop lid at night?
wry_xy 3 hours ago|||
The self-hosting crowd, Linux fans, OSS contributors, etc, are our own worst enemies.

If we care about self-hosting and open source and decentralization because we believe it leads to a better world, and not simply because we want to be control freaks and snobs, then we need to learn how to build shit that is actually convenient and usable and understandable to normal people.

I think it's possible. I think many people would want this, and care, if we could package it up in a digestible way.

lilbigdoot 3 hours ago|||
Linux has been more user friendly and accessible to me for normal use than Windows for years now. I really do not understand this comment
guhidalg 3 hours ago||
Talk to people not in tech.

Windows never ever ever makes you open a terminal to do anything, that’s scary to non-tech people. Windows “just works” and most people just want to minimize their computer use for work and go watch tiktok on their phone.

rangerelf 2 hours ago|||
"Windows just works" is such a lie.
luqtas 2 hours ago||||
which scenario someone needs to open terminal if all they need is a browser and an office suite?
drdaeman 2 hours ago|||
Windows makes you open a terminal just to install it properly.
pelagicAustral 3 hours ago||||
I'd say one of the major frictions with self-hosting is that for the most part it does always going to take a lot of tinkering around and maintaining things going. A world were we have a lot more people self-hosting would be great, but as it stands, it's an utopia, because normal users will get annoyed the fuck out of the environment the second their infra is busted because of any of the million reasons things break.
lisplist 54 minutes ago|||
Self-hosting has gotten substantially easier with LLM assistance. Now instead of endlessly troubleshooting problems, I can just have an agent do it while I go do something else.
pydry 3 hours ago||||
Self hosting is in pretty much the same state smartphones were in before the iPhone.

I remember trying to install apps and it was annoying for all the same reasons self hosting is these days. It was niche, only hobbyists did it, "nobody cared" and the presumption was that for normal non-nerds, phones were exclusively for calls and texting.

There are some spectacular successes in terms of the actual self hosted software (immich and home assistant) but the overall experience is still N95 not iPhone.

The part where self hosting falls down isnt the apps it's config, networking, reliability and backups.

japbujan 3 hours ago|||
[flagged]
davedx 3 hours ago||||
Ollama and friends are pretty damn easy to use already. Hardware is the real barrier
lurking_swe 3 hours ago||
let's see here, you already make a crucial mistake that a product manager wouldn't. (I say this as a senior software engineer that has worked with product managers a lot)

If I tell my mom to download and install Ollama on her Mac, do you think she's gonna have any clue which model to download from all of the 5000 models she can choose from? Do you think she understands which ones will even work well on her machine? What if she downloads one that's highly recommended online but it doesn't work on her 64 GB MacBook (hypothetical example)? Because it's too too large/slow...

and can Ollama, out of the box, do actual work on her computer, similar to the ChatGPT app with its "Work" mode, and dozens of built in connectors (google, etc)?

you call this easy to use for a normal person? You just made their point. The ChatGPT and Claude apps that my mom can download, set the standard for her. Her standard of good/easy is not what you and I as nerds find easy. you need to really put yourself in the shoes of a tech illiterate person. we haven't even touched on the fact that Claude and ChatGPT ship with easy to use iOS and Android apps that sync up perfectly with her Mac.

anonymousDan 3 hours ago||
This is a bit of a strawman argument. Who cares about your Mom (no offense)? The important part for me is that technical people (e.g. software developers etc) have access to competitive open source and/or local models.
guhidalg 3 hours ago|||
That may be personally important to you, but the market for tech-literate people is dwarfed by everyone else.
graemep 2 hours ago||
In terms of work usage its the technical people who will install it for other people. "everyone else" does not know how to set up a server or install an OS etc. its all done for them by someone else.

The consumer market for LLMs is small. compared to the business market.

petra 2 hours ago||||
What percentage of techies use self-hosting for a significant part of their services?

Before that number is high, can we realistically expect non-techies to use self-hosting?

forshaper 3 hours ago||||
At some point there is a trade-off between those far away controlling "your" stuff and convenience. It is much easier for someone to make something convenient if they have control over it. For example, troubleshooting when you own the stack versus troubleshooting when you don't. However, if someone wants someone far away to help them troubleshoot, then it will be easier if the one far away controls the stuff. There is a trade-off there.
goda90 3 hours ago||||
The problem with open source is that it is often written by people who are fixing issues they have or making features they miss. To get to the point of contributing, you have to go through the initial setup process. Once you've done that, it's not longer a problem you care about, so you're less motivated to contribute code to make that process easier.
nemomarx 3 hours ago||||
There's a decent opportunity where AI coding agents seem good at creating a very "average" and understandable UI. Maybe not perfect, but probably better than Gimp?
zoeysmithe 3 hours ago||||
Its worth mentioning "normal people" take hours upon hours of classes and trainings to use typical office productivity software and other job-related software, on top of watching and learning from tutorials and family and friends to use personal software and websites and apps. And heaven knows how many support calls and tickets to just do what is supposed to be "easy" in the closed source world.

Also, none of this is intuitive and a lot of popular closed stuff is harder to use (and vastly disliked by the very same normal users) than some popular open source stuff.

I'd even mention 'normies' want self hosted Word and Excel and regularly reject the web-based versions of them. Normies don't want streaming games but a gaming PC or console. Normies may buy streaming services like Netflix but get upset when their favorite shows are removed, so they actually don't like netflix. Normies complain about censorship on twitter and reddit and discord and would love an alternative they control, but may be unaware that's an option and seem stuck on those services and hate them.

Normies want self-hosted all the time too. Their ability to understand and do this is what's lacking. So that's a conversation on its own, but normies aren't actually happy. Form what I can tell most people hate 50% of the stuff they use right now and most of those complaints revolve around the cloud-hosted/lack of control/privacy and other issues self-hosting or federated services or FOSS apps could arguably solve for them.

This is a big conversation but imho tends to boil down to who has the marketing and sway to get you to install or use something, not that typical end users are savvy judges of software and useability working purely on merit and informed facts. Does your grandma actually like Facebook, or is she just lonely and its the lowest friction way to not be lonely? Do you really like Chrome or Word or is it just easy to get and popular? Is the big mac the best burger or is just easy to get and popular?

betterunix2 3 hours ago||||
> shit that is actually convenient

I mean...sure...but how convenient was that Office 365 outage? How convenient is it to find that a feature you actually depended on is gone because the company that runs your software decided to get rid of it?

It is always a back-and-forth. It is not as if centralized computing is a new idea, in fact it was the original way to use computers and people found that issues of control mattered and switched to PCs; the pendulum swung back because people realized that there are advantages to centralization (e.g. not having a failed hard drive destroy high-value documents). The cycle will probably continue for a long time to come because the advantages and disadvantages of both approaches are fundamental.

bendergarcia 3 hours ago|||
That inconvenience is still less inconvenient than not being able to figure out your dns or vpn or tail scar settings or why it won’t render and why the machine isn’t reachable. When running your own services is easier than it currently is and lot of tools are built around notifications and debugging that are actually understandable, then you’ll have more adoption of open source software.
coldtea 1 hour ago|||
>I mean...sure...but how convenient was that Office 365 outage?

One outage in a blue moon? Bloody very convenient. People have more outage days per year on their own laptops.

templar_snow 3 hours ago||||
This. The FOSS/etc crowd don't even try to bother with outreach on tools that the majority of people would want if they were marketed correctly instead of kept as niche projects for people to be smug about. Consider the EU rolling out Matrix in a huge way, or people moving to Mastodon. For most normies, "Use this to escape Elon Musk" is more than enough sales pitch.
pyrale 3 hours ago||||
> not simply because we want to be control freaks and snobs

It’s not so much about being snobs as it is about not having time and energy for a second job. The level of quality you demand simply requires a level of attention, investment and involvement that can’t be provided without working full-time on it, for profit.

Telling a bunch of hobbyists that they should provide service of a quality equivalent to that of a trillion-dollar corporation if they are serious about it is gaslighting.

dotancohen 3 hours ago||
I hate to say this, but Claude Code has turned my home projects and personal scripts from "I tolerate this because I need it" to "This works so nicely I might make it available to others". AI could really help here.

I'll go purify myself now.

agile-gift0262 2 hours ago|||
> we want to be control freaks and snobs

That's a mirror that hurts to look at that you just held in front of my face. Guilty on both counts, even though I try to convince myself I'm not

nsbk 3 hours ago|||
Hard disagree. I'm running Qwen3.8-Flash-Next using Strata on a Mac Studio and it covers all my needs. One drag and drop install. Nothing like self-hosting your email.
mortalapeman 3 hours ago||||
Seconding this. If you know how to build with a dumber model it scales crazy well. And Qwen 3.8 Flash Next is just wildly good even at low quants. I'm trying to find it's limits and I've been using it for over a week building and haven't hit a wall yet.
choxi 3 hours ago|||
Are you using it for coding? I'm looking for a good coding model to use for Xcode and NextJS apps
3836293648 1 hour ago|||
Really? I hit a wall immediately, though I'm using it to implement things originally written in a proof language, and wasn't that good at prompting to begin with.
paxy 2 hours ago||||
What are you disagreeing with exactly? Qwen 3.8 flash costs a few cents per million tokens from cloud providers. You can get effectively unlimited usage on a $6/month plan. You were okay with buying a $5000+ Mac Studio for it because you have the money, but you are in the tiny minority.
nsbk 1 hour ago||
I disagree with your statement saying there's no difference between self-hosting local AI and self-hosting regular software. Regular services (which I guess is what you're referring to) require uptime, to be accesible from the Internet, backups, security patches, the list goes on. The main argument against self-hosting has never been cost, it is a complex task that requires knowledge and dedication. On the other hand, you can self-host on a Raspberry Pi or ancient laptop for close to nothing.

A local AI can be as simple as a program that runs in your computer [0]. Any machine with 12GB+ of VRAM can run the same model I'm using. I has a chat interface, can see images, control your browser, generate artifacts, work on projects and workspaces. You name it.

[0] https://stratallm.org

xandrius 3 hours ago|||
How much was your Mac Studio?
nsbk 1 hour ago||
This is not about cost. Any machine with 12GB+ of VRAM can run Qwen3.8-Flash-Next at usable speeds using Strata
__MatrixMan__ 2 hours ago|||
> But 99%+ of consumers aren't going to care.

They don't now, but they might soon.

It used to be that bad behavior on the part of our service providers was annoying because you'd get ads and spam. Nowadays the leaked data gets aggregated into apps that help government spooks abduct your neighbors. The effort remains constant, the threats it helps you avoid are getting more dire. Eventually those lines will cross.

bee_rider 3 hours ago|||
I think ultimately local LLMs will win but only because models will get good enough that the “cost” of running locally on a 2030 chip will be less than the “cost” of talking to the network. Especially given that building wider/lower precision LLM accelerators is a perfect task for hardware companies (embarrassingly parallel in an era where sequential performance improvements have long dried up).

In terms of self-hosting vs remote-hosting regular software… I mean, a lot of regular software is fundamentally about communication (social media and office suites) so that goes on the server. LLMs don’t benefit much from being remotely hosted, other than that it makes the flops run in somebody else’s machine. But that will be hard to fund in a universe where the upper-limit to what can be charged is defined by how much additional cost “good AI hardware” adds to the price of your phone. We’ll see I guess.

jbjbjbjb 3 hours ago|||
It’s more about self-hosting the hardware which is more desirable than thin clients.
wmichelin 3 hours ago|||
Self hosting is not free, it's actually pretty hard to setup and keep running sometimes. Time and cognitive overhead are money!
Zambyte 3 hours ago|||
> cognitive overhead

I told a remote model to set up strata on my local machine. With the best options for my hardware. If someone can figure out using remote models, it's trivial to bootstrap local intelligence assuming available hardware. The hardest part was knowing that strata exists.

Ninja edit: I've even had my agent tell me about suboptimal hardware configurations and told me what to do to fix it (XMP settings, having hardware in suboptimal slots). Cognitive load is simply a non-issue. The only issue is time, but in the grand scheme of things, it's a trivial amount of time to get up and running.

mortalapeman 3 hours ago||||
Just get a 5070 and 64 GB of RAM and download Strata. It's the ollama of actually usable local AI.
nullsanity 3 hours ago|||
[dead]
barbazoo 3 hours ago|||
It's so sad that we are not empowered to act as more than just consumers as if it was some kind of shared religion. How do we get out of this cycle?
random3 3 hours ago|||
People complaining about how difficult self-hosting is seem to be stuck in the past and missing the point. It's the very AI that we're talking about that removes most if not all friction points related to software configuration and maintenance.

While this doesn't remove the upfront cost (which could be fixed with basic financing like Apple does) and physical manipulation (e.g. replacement on failure, etc.) it does lower the barrier.

Finally, there's another path I find interesting and interested to think more of which is sharing hardware locally, from your house to your neighborhood, etc. We've done this in the late 90s - early 00s with Internet and it worked really well. I think it could work great for compute.

I see a lot of opportunities in this direction, actually and most are actually enabled by the latest improvements in AI, so most of the "old" thinking doesn't apply anymore.

bobajeff 3 hours ago|||
That's like the argument on why people don't care about privacy. Just because most people choose that, all things considered, they are willing to sacrifice sovereignty over their own computer usage doesn't mean they don't care about it.

By that reasoning you'd also conclude that no one cares about affordable housing or having a good work life balance or having your own lake made out of solid gold.

redanddead 3 hours ago|||
It comes down to the incentives

Think of who uses local AI, it’s either

-power users upset with the limits —-cost of intelligence is going down

-people and corps that want stabiltiy —-a massive self own by the labs, this is justified

-American labs seemingly snitching on their users (wtf?) —-huge incentive for people to use Chinese inference, which is cheaper along with other benefits

Regardless, we’re talking about a subset of a subset of people, 1% of 1%. The big bet is that everyone will become AI power users. I think it’s accurate, forget openAI, fuck Anthropic, let’s look at us, the people. Entire workflows like coding have been buried, there’s no going back to how things were before. Is it reasonable to say that will happen to the other work streams? There’s no indication not, and it’s a total gamble if it’s yes

lrvick 2 hours ago|||
This is like owning a car vs using a train. Both are valid choices depending on your goals.

I really do not care if everyone does it my way. What matters is that the choice exists for those that want more ownership, freedom, and privacy.

That said, most next gen laptops will be shipping with capable inference hardware more than most consumers will ever need. Not using the GPU people have in front of them would be silly, but people are silly.

jmyeet 3 hours ago|||
Here's one difference: currently lawyers using cloud LLMs violates attorney-client privilege. And you have to be really careful sending confidential medical information to one so as to not violate HIPAA.

More broadly, those prompts and the responses have value. There's a whole shadow economy in China of aggregating queries across Claude (etc) and then using that as training data. It's like your entire search history is for sale.

Clould infra has typically been opaque to the provider. You run an MySQL instance on an Amazon VM and Amazon doesn't have easy access to the unencrypted traffic and what you store. Cloud LLMs don't operate that way.

AI hardware is improving at a rapid pace. By 2030 I believe a significant amount of LLM usage will be local for these and other reasons.

jimbokun 3 hours ago|||
Enough that Apple currently build enough Mac Minis and Mac Studios to satisfy the demand.
adammarples 3 hours ago|||
I don't know about this whole argument that hardware will become obsolete. If it can run Opus 5.5 today it will always be able to run Opus 5.5 until it burns out, and the quality of the models it can run will continue to increase as the software improves.
nullsanity 3 hours ago||
[dead]
herf 4 hours ago||
I love local AI, but even so, I admit the economics point the other way. Unlike gaming, AI is a uniquely good match for data centers because the applications today aren't latency-sensitive at all (except perhaps DLSS/self-driving/robotics). Most importantly, utilization is massively better with always-on servers doing parallelism (in-flight batching). Also in some areas (like California) our local electricity is quite expensive, so moving the work 100ms away can help. The privacy implications are horrifying, though.
younglunaman 3 hours ago||
I tend to disagree here. Think about how useful it would be to have a nice llm in real-time debugging your applications loaded entirely on your computer. That sounds pretty sweet to me. I don't see how that's not really cool / immediately useful to you?

Maybe your point is the majority of the time that's not needed, fair, then maybe the approach is both?

herf 2 hours ago||
Well, LLMs handle raw data perhaps a million times slower than conventional algorithms, so in "bytes" and "latency" terms, a dialup modem can handle most AI communication today - a rack of cloud GPUs 100ms away is still faster for a lot of things than $10k of hardware in your GPU, Mac, etc.
drdexebtjl 3 hours ago|||
> the applications today aren't latency-sensitive at all

This is circular reasoning. We don’t have latency-sensitive applications because sufficiently powerful local AI is not profitable to entertain such applications.

herf 2 hours ago||
Absolutely, and there are applications like self-driving or robotics or surgery or whatever where it's critical to have the work be local and never have a network hiccup. But these are all in some sense new markets, though I guess these would be growing faster if we had abundant local AI. Right now most of the AI applications that people pay for are ones that make human workers more efficient, and humans are already high latency.
musha68k 3 hours ago||
The economics point the other way because capital is aligned against you. Should this be a bubble that pops we will hopefully see prices come down again.

MooresLawIsDead and his spidey web / senses thinks it already popped:

https://youtu.be/YJeK2gumpEI

brendan_rules_t 3 hours ago||
I have a 128gb ddr5 unified system, that i have been setting up with linux and local llms and other models and i have to say it feels beyond next level! I still use claude, grok, codex, and neo clouds to offload heavy work loads, but anything i want private or says too much about my system is now done locally. The voice agent experience is incredible
afzalive 1 hour ago|
what models are you using for which purposes and what is the voice agent stack please?
montroser 3 hours ago||
I run Qwen3.8-27B Q4 on a $1500 box in the basement with 32GB VRAM, and a customized OpenWebUI on top. It's pretty great. Occasionally I consult Claude for a second opinion, but I'd say I've moved 95% of my previous conversational Claude usage to be local instead. I use it for leisure coding projects too, but for real work, it's still DeepSeek 4.1 Flash for now.
cj00 3 hours ago||
Invariably when local AI comes up, people try to see how it can be used for frontier-level AI tasks like asking it to one shot a web app from scratch and then they get disappointed when they discover it's not performing at the same level as Opus.

But they're ignoring that a local LLM can still be very useful for any kind of repetitive task, things like categorization or generating embeddings, translation or summarization tasks. I'm self-hosting Karakeep and use a local LLM for the AI tasks like embedding.

And once you've got that local LLM running, you can start thinking about integrations with your own custom built apps.

sachaa 3 hours ago|
Also, they are thinking of one single LLM model at a time. What if we have a network of dynamically loaded micro-models loaded on demand based on the user intent? Think of a global DNS of micro-LLMs. We need to decentralize AI. I started building this here https://modelxare.com
smath 3 hours ago||
What is a good place to track the latest capabilities of open source local LLMs vs commercial models? Things like overall intelligence performance, latency/speed, hardware needed, cost.

Anecdotally, I believe the open weights local models are constantly catching up with commercial ones, but I've honestly lost track. This is a dynamic space.

MoreThanMe 36 minutes ago|
[flagged]
1970-01-01 1 hour ago||
Voice interface is the "next big hit" as it is removing the very last hurdle to accessibility. Anyone that can talk and hear will be able to use AI. No screen, no keyboard, no problem. This was supposed to be the home assistant in a nutshell but we all know how limited it was in reality.
layer8 1 hour ago|
No screen is limiting though. Imagine if any form of written communication or written record were prohibited. We want the AI to integrate with our vision bandwidth. Similarly with using our hands for interaction. There are many tasks that are much more efficiently and accurately done with our hands than by spoken instructions.
sachaa 2 hours ago|
Why should users have to choose a model at all? We don't choose a codec before playing a video. An app should be able to request transcription or summarization, and the browser should find and run a suitable local model. That's the layer i am exploring with modelxare.com.
More comments...