not saying i disagree with the general statement, but there need to be options, not everyone has a machine capable of doing the same type of lifting required to properly run a local version. so what, if my machine is older i'll be locked out? restricted? forced to pay?

- can we get suggestions from people on what would the equivalent for android

- and for the web / javascript / svelte applications?

- suggestions for local OCR for bulk images?

I‘m surpised at the presented dichotomy between JSON formatting and what the Apple SDK provides to parse output into structs.

Based on what I understand about how the former works, I would assume that the latter has the same properties and failure modes.

> “AI everywhere” is not the goal. Useful software is the goal.

Great observation! Often the excitement of novelty makes us lose sight of the real goal

While I agree that would be the goal, we are too early for that. Just like how speech recognition used to require many server in a Datacenter to process and you send your data over. It is now completely on devices.

We are at least 5 years away from that. And DRAM needs a substantial breakthrough in cost reduction.

People are trying to “make the best software”, though.

I think the Quixotic accelerationists of AI are more or less a vocal minority of the people who make software, and the choice of online APIs over local systems is largely a choice made for users, rather than developer’s laziness.

You can do more and better with private AI today than with local models. There is no getting around that. Even if local AIs get better, being on the cutting edge of LLM performance is often a very worthy investment.

Most people won’t settle for a product if it’s not the very best and incredibly convenient. That’s a high bar, and local AI often doesn’t meet those standards.

HN’s insistence on treating all users like they are open-source, privacy-first, self-hosted Linux fanatics is painfully corny.

Local AI is definitely going to be the future as these models continue to advance at the rapid pace they already are.

This is why I believe OAI and Anthropic I’ve been so aggressive at offering services outside of their pure models like Claude Design. This is what will be competitive and keeping people subscribed.

Opus 1M context window and lighting fast response time is hard to compete with, even if you run a local A100 the local models are just not as good as tool calling, long running tasks and non-hallucinations

The roadblock to this is you seem to have to build it yourself. I've noted that none of the current cloud models are very good at building a replacement for themselves, and there's significant work that needs to be done to make a local LLM reliable in any way. I haven't found a single standalone package that makes setting them up easy. Sure, I can run Hermes Agent and a model, but getting the self-reflection loop in and all of the other stuff the need to actually be good? I'm still at it, trying to get anything to work reliably and factually.

I'm building a protocol and router runtime for hybrid local/cloud AI.

The goal is that you would assign roles to models based on tasks, capabilities and observed performance. The router would then take care of model selection in the background.

It's tricky though. Probably have another two weeks before I can release the runtime.

I have a preview up at https://role-model.dev/

You can follow me on Twitter if you want updates (see profile)