What about local models do you find preferable?

I guess "starting to find them preferable" suggests to me you think they work better, but this is surprising to me so I think I may have misunderstood, so I ask!

Like you're saying they work better than the proprietary models (in what ways?), or you find them mostly good enough and prefer the privacy or cost, or what?

From a privacy perspective, your objective is to stay away from people who have interest to snoop on your conversations.

So from the perspective of your teen, they would benefit from using z.ai or ChatGPT or Claude, etc, rather than the local server where you can see all the conversations.

What uncensored model do you recommend using ?

What is an "ephemeral" model in this context?

How many tokens /sec?

> I run an uncensored, ephemeral model for my own use and it's an entirely different experience than anything you can pay for.

Dont. Goon. To. LLMs

Only if you let them.

How could it not? I get the whole fear of AI making robots and going anti-human, but after using the tech for a few months now that seems too absurd.

And we are definitely not going to put our users on a watch list DB and send their data to the government?

And how do we prevent Chinese companies from training on our open AI models and offering their models for free?

Maybe one of those trillionaires could help for a bit before leaving to make his own AI model, too.

i’d really love to be wrong, i don't think that the economics of it would let it happen.

the potential of wealth creation with AI is so high, and also the fact that research, pre-training and inference is so expensive that, that any open-AI would eventually become OpenAI.

We could all chip in

[flagged]

There is an understandable gap between the capabilities of closed models and those of open models. The current difference is primarily expressed in the cost of hardware necessary to sufficiently run a exactly comparable model. A single higher end graphics card running on your average gaming computer, is capable of running small to medium models that compare with those of their lab-born counterparts in the small-medium range. But the heavyweight models are still outside the realm of possibility for all but the most well-funded individual.

However, I would highly suggest more people experiment with these smaller models. They are incredibly capable in many ways that many people dont realize.

The perceived capabilities of the larger models are also much less the result of the model having more parameters/training cycles, but rather that they are being run through well-made harnesses, something which the open-source community is rapidly approaching with near-peer solutions of their own.

In short, much of the gap between between open-weight models and the larger proprietary models can be considered more of an issue of perception and not an issue of capability. There is a fundamental gap economically, but not so much in capability. The open source community is rapidly closing the gap on these larger labs, especially thanks to the amazing research being freely given openly by well funded chinese labs.

See my comment to parent. I've been using local LLMs for practical, personal tasks for a few months now very successfuly.

You can run fantastic local models if you have either:

- M-series Apple device with ideally >= 24GB of VRAM

- RTX [345]090 GPU

I'm fortunate enough to have both and use an M-series laptop as basically a persistent server (I don't use it much and when traveling typically just use my work laptop). My desktop doesn't act as a persitent server but I fire up llama.cpp on it all time for quick chat sessions.

If you have one of the above devices and can dedicate it as server there are additional layers of tooling you can use that dramatically improve the experience. In particular Open WebUI allows you to add tons of useful tools (image gen, web search, code eval, etc), and agent harnesses like Hermes can make the current gen small models very capable. I have an agent in chat on my phone that basically handles all the sys-admin for the server it runs on.

I'm also curious, specifically about the cost of training vs inference, and comparing that to other industries that can have high R&D costs. My instinct says that open weights aren't feasible because of the obvious issue where there is no incentive to develop your own model rather than just taking someone else's model. However, I could see a scenario where a hardware company designs a model that is open weights but optimized strongly for their own proprietary hardware, cutting their costs of inference low enough to be competitive with a hypothetical other company that doesn't have any R&D expediture.

Sort of. A full trillion-parameter model needs about $300k of server hardware to run in and a lot of electricity, making it feasible only for very wealthy individuals, but quite practical for businesses and institutions above a certain size...although they in turn would typically gatekeep access.

You can drastically reduce the requirements by running models at a lower bitrate, which somewhat reduces accuracy but not that much - think of the difference between an MP3 vs uncompressed audio. With this and other tricks, you can get high end models down to a size where they can be run on a high spec desktop workstation affordable by an individual or small business.

Obviously I'm heavily oversimplifying here. I think a useful parallel is to consider situations from the past where you would once have required corporate budgets equivalent to the price of a house to run a large database, but over time it became accessible to anyone with the requisite expertise and relatively affordable hardware.

It depends entirely on what you want to do and think is feasible. Small models can almost certainly run on the computer that you already have. They can do good tool calling.

Yes they are you can use Qwen, DS4 Pro and GLM 5.2 if you have the hardware to do so.

They are not SOTA in various ways but they have better economics.

If attractive, cloud providers could develop open models with their own investment, and sell hosted access as a business model. While Google checks these boxes, I haven't seen a Google much marketing focus upon their open models (Gemma) coupled with hosting. groq could conceivably train its own models, but groq's business model hosts open models (GPT OSS, Qwen 3, Llama 4 are currently their prominently advertised models on their site... which seems out of date to me) trained by other organizations.

I hope/wonder if it will go the way computers did. We may learn to more effectively build RAM or parallel compute, and use it more effectively, in the coming decade in such a way that we can democratize more and more like we did with processors to the point that they're ubiquitous.

Is this satire? I really can't tell.

>This is a serious competitive advantage

Given they have laughable uptime and I have yet to find a useful project mostly written by claude... I doubt it.

EDIT: Parent commenter completely rewrote their comment while I was replying. I'm leaving this up as is. The text below is their original comment that I was responding to.

> What are you pushing by pointing out not only in this thread but the previous one too, quite in depth, that it's not new? I know you can claim you're just a stickler for accurate reporting but you seem really invested

Because these threads degrade into panic with the assumption that everyone is getting ID checked now.

Pointing out that the policy has been in place for 2 months puts it into perspective that this isn't a sudden policy change requiring ID for everyone. It's been this way for months.

If they roll out mandatory ID checking for everyone, that would be a different story.

If you have an actual rebuttal, present it. Don't use thinly veiled implications that OP is some sort of a shill as a substitute for a cogent rebuttal.

You could say the same about China, where only 7% of people can vote. In the US, 72% of people can vote.

These things are always a slippery slope. They rarely, if ever, achieve their safety goals but they almost always achieve the goals of the corporate interests to garner further data for advertisers and increase surveillance of the populace by the government through proxies that buy said data and then sell to the government.

I think the general assumption is that it only aids in surveillance.

It is not merely showing proof of age

It is using the proof of age requirement to require a much larger ask -- full proof of identity

Age verification could be done with any of a variety of mathematical systems showing you have a proven age-valid ID but not revealing your identity. But no one is suggesting they build and use such a system.

Why is that any different? What’s to stop that company sharing information with the government?

with sufficiently many public/private partnerships, what's the difference?

The point is that you're all shitting on China 24/7 while not recognising that you're slowly but surely building something very similar at home right now

> free speech, civil liberties, voting, are in China all well below the standards of the west

Haven't heard China deporting students for criticizing foreign state or electing people involved in p*dophilia and then closing eyes for those crimes

Also the Chinese social credit system is not what most think it is - https://en.wikipedia.org/wiki/Social_credit_system#Misconcep... .

Because we dismantled class and caste. It’s a “meritocracy caste” system (nevermind that you likely get huge advantages based on your origin)

It’s funny, 30 years ago the argument was the exact opposite: China opening up and doing business with the rest of the world would force them to liberalize.