distributed AI computing so your hallucinations can be geographically diverse too
whatjustin 1 day ago||
The real test is throughput. I'd like to see tokens/sec at higher concurrency and with uneven hardware.
darkpicnic 2 days ago||
cocompute.ai is already doing this really well.
SwellJoe 2 days ago||
Is it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).
darkpicnic 2 days ago||
Good point. I know cocompute is working on splitting, but it’s not there yet; I was referring to the round-robin delegation within a trusted pool. Mesh LLM looks great too!
dnoberon 2 days ago||
Cool, always good to have more in the ecosystem. I love Iroh and hope this continues to succeed.
Onavo 1 day ago||
Is this truly more secure though? The host can still see your data.
I knew this was possible, i asked chatgpt about a year ago and it said no the latency would be too big of a problem. I spent the best part of a year learning libp2p and was looking for a project to do with it at the time.
nullc 1 day ago||
Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.