I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.
stymaar 12 hours ago||
Throughput is not a problem as you just share relatively small vectors (a few kilobytes in size), the key issue is network latency.
i386 20 hours ago||
That isn’t true. llama RPC is incredibly slow but staged splits in skippy are orders of magnitude faster.
vigsterkr 10 hours ago||
the https://query.mt/ project has been using iroh based mesh for a while. maybe give it a go, especially if you wanna use your mesh models on your mobile phone as well.
derdi 2 hours ago|
They might want to make their docs less confusing. https://query.mt/showcase/gpu-sharing/ says "no API key needed", and talks about running llama.cpp on a friend's GPU. No API keys are needed for this. But then the page goes on about who "holds" API keys, and where API keys "stay", where API keys are "locked", etc.
darkpicnic 21 hours ago||
Does Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?
tekacs 21 hours ago||
I'm not affiliated, but yes – the main 'point' of iroh is that it's 'dial-a-key', QUIC with encryption based on the keys of the endpoints.
metadat 19 hours ago||
Just wondering, why do you care about encryption in this context?
darkpicnic 19 hours ago||
If payloads to LLMs are being passed around to various nodes, even trusted ones (like friends and family), it gets awkward if you send something very personal. Think sending a medical question to medgemma:27b.
oezi 18 hours ago||
Even if transport is encrypted, the LLM computations will always be clear text, right?
embedding-shape 11 hours ago||
Indeed, it's in-transit-encrypted so snoopers won't be able to see it, but it's not E2E encrypted nor in-process encrypted, the one's doing the inference could obviously see the input/output.
SubiculumCode 13 hours ago||
All these ASICS being designed and specialized for AI but none seem to be being built for consumers. Reason?
josefrichter 13 hours ago||
Is there a catch? If not, this would be super useful.
stymaar 12 hours ago|
The catch is that the token generation speed is going to be limited by network latency, making it unbearably slow to run over the internet.
It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).
turtleyacht 22 hours ago||
It sounds like iroh enables distributed compute without having to finangle custom hardware.
luciana1u 13 hours ago||
distributed AI computing so your hallucinations can be geographically diverse too
Onavo 7 hours ago|
Is this truly more secure though? The host can still see your data.