Posted by ar-nelson 1 day ago
> Up to 7.4 tokens/s.
It's not really "Flash" if it is 7 tokens per second.
Also don't bother with any GPU lower than an RTX 5080 or even a DGX Spark (it is a scam) if you are seriously considering running frontier local models.
Strata runs considerably faster than that on similar hardware, though this one doesn't seem to be a fork.
The problem is that this particular model is so cheap, compared to others. Optimizing to host it locally with relatively poor performance just doesn't make sense when yesterday I burned ~180M tokens for $2.20.
Is that faster?