Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.

Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.

Yet, seems like there is still another year for improvements.

I like local models (but not mainly using them) for offline needs.

For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).

I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].

[1] https://arxiv.org/abs/2507.13575

If true, that would be a very welcome development.

That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.

Also, perf speedup?