Posted by ascorbic 3 hours ago
However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Secondly, the evaluation suite they use to claim that FP8 KV quantisation is indistinguishable is noticeably lacking coding benchmarks; in long-running tasks, minor tool call errors compound over time.
> None of this would matter if it changed the model's answers
If they want to assert that the answers don’t change, then perhaps they should calculate the statistical distance between the token probability outputs or something to that effect. I doubt the results would indicate that the answers don’t change by any reasonable interpretation.
Maybe the results are still good enough.
Why… I wanted to see if it’s worth it to use cloudflare’s endpoint but I can’t even see the pricing
I love AI, but I really hate reading it.
How well it would work on this site, I'm not sure.