There was an article on HN a few weeks ago where someone detailed how they managed to get an old datacenter GPU to run in their consumer PC, getting decent performance with qwen. He spent something like $200 on the GPU (second hand of course).

So yeah, I think models on local hardware will be quite common soon among the tech savvy (such as people creating software).

Indeed, and with some tinkering around the harness it can even punch way above its weight.

> You need a system with either 32+ GB VRAM

I do hope you're right that it will get cheaper over time (it should), but right now 32GB of VRAM is not affordable to a lot of people. You're talking ~$4500 just for the GPU, or $800 ish used if you can find one.

And a unix workstation in those days could be high 4 or even 5 figures, depending on configuration.

I thought Moore's Law came to an end in the last decade?

Certainly the transistors/chip or transistors/$ or flops/$ have not been progressing at the same exponential rate as during 1970-2010. There is still progress, but it's rather slower.

IMO the more useful distinction is in analogy to VPS versus SaaS/PaaS. Open models allow you to use any inference provider you like, including local ones, similar to running open-source software using VPS providers. You’re not bound to a particular SaaS/PaaS as you are with closed model providers. That same freedom also allows you to self-host when you care about that.

Software models and hardware are getting better all the time—and that’s where some big companies spending billions might stumble! In fact, Microsoft recently announced that they’re scaling back a bit on their AI investments.

... or rising, at least as long as there's a RAM shortage.

Yeah. I would expect that Samsung or Apple’s bundled AI things (whatever those end up being) will be good enough for consumers, and open models will be the things that actually get used in the products of the future because they will allow technical people to alter models in ways that specialize them for whatever use case they’re building for. They will also be the onramp for engineers/data scientists of the future, and thus will have more available, specialized knowledge over time.

The bitter lesson would claim that we only need one model with more data thrown at it, but I’m a bit skeptical that we’ll end up with “the one true way” of building a model, and I think there will be model tradeoffs that we can pick from as the industry matures.

and that is millions of people; hence the use of the word 'us'.

Companies aren't really that better. Actually, sometimes it's even worse, because they are not even formally required to serve the public.

A drop in the bucket compared to the value of the collective human work that was stolen to train it.

edit: come to think about it I think the ratio of one drop to one bucket is vastly over estimating the ratio of the trainer's effort.

They got paid. That’s what the money was for. It’s the investors who backed these foundational model companies who will hold the bag as more open source models come along and consume more market share.

Some consider copyright to be unethical. "Information wants to be free".

Paying them may now be impossible. There might be some legal settlements still.

Preventing a handful of massive companies from continuing to be the only ones able to make money off that, not only unimpeded but with overt or covert state assistance (regulatory capture, ownership, whatever), at least puts an end to the worst of the abuse.

If we have broken the idea of copyright, and we do indeed appear to have broken the idea of copyright, why should trillion dollar companies owned and controlled by strange or psychopathic weirdos and their circle of investors be the only ones benefiting? Why do Sam and Dario or the US government get to decide when and for whom the tap is turned on?

Not too mention the unbelievable cost of actually doing all that training.

Are the going to pay for the societal harms they cause?

Weights are meaningless if you can’t run the model from a computer under your desk.

I'd love to see credible numbers on the energy usage of thousands of people running models on their own devices compared to sharing data center resources to run big models that serve many different people at the same time.

My hunch is that the energy/water usage of the data centers is a whole lot more efficient than everyone running at home, but I'd be interested in seeing real data on that.

NO!

This is the wrong approach that will turn us into serfs. We need big honking models that do what the leading foundation hyperscaler models do to within a few percentage points of measured performance.

The small-scale models are not productive, and the duct tape solutions built on top of them are hobbyist-tier "year of Linux on desktop" toys.

I imagine fedora-wearing, crypto-shilling, coupon-cutting boffins every time I see small weights thing lauded as the future. This is the Pine Phone F-Droid of AI.

"SMS works most of the time on my phone, I swear! I don't really need my banking app!"

That is not big model energy.

Nothing outside of the top ten is worth spending any time on, and we need to focus on models that bridge the gap.

You're talking about impractical toys for highly technical people wasting their own time. That doesn't move the needle or have any economic impact on the competitive landscape.

We need sharp teeth that bite at the legs of the top-tier foundation labs and hold them back from running away with the prize.

We've been through this time and time again over the last thirty years. It's the same shaped problem as before. We don't need toys - we need real infra for real people paying money to do work. Not freeware for freeloaders who don't spend and invest in the problem space.

Large models fit that precisely, because it forces investment into a wide variety of open infra, routers, inference engines, etc. Not to mention the weights ecosystem itself.