Posted by philipkiely 5 hours ago
I would define a "frontier model" as offering the highest degree of intelligence at any cost, or without regard to cost. The frontier today is clearly Fable/Mythos, with the "efficient frontier" at Opus/Sol.
As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.
Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.
Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].
Everything old is new again (:
[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...
[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf
[2] https://www.microsoft.com/en-us/research/publication/distrib...
That's because it's just "guess and check" not some deep universal insight.
The images start out looking like this: https://philipkiely.com/images/blogs/how-to-write-a-book/des...
the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there
But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.
Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate.
> In practice, the efficient frontier is very jagged. Rather than a smooth, continuous line between outcomes, small changes can have big impacts. These cutoff points are often unintuitive and must be discovered empirically through sweeps.
> However, quantization introduces a new set of tradeoffs between quality and serving efficiency. This is a particularly jagged frontier, where a large degree of improvement to serving efficiency is possible with little-to-no reduction in model quality, especially when using microscaling floating-point number formats like MXFP4 and NVFP4.
Would appreciate ideas on how to explain in greater depth
The efficient frontier of LLM inference is a line, not a frontier.
this is a frontier:
https://upload.wikimedia.org/wikipedia/commons/e/e1/Markowit...
no matter how good is something a smart person writes down, a pleb will come along and try to hang on its coattails.
If you want to steal an idea for this, steal indifference curves, they'd make more sense.
