ternary is totally losslessly compressed anyway.

Why not just use an 8-bit LUT to encode the 256 most common ternary vectors with 6 components. That means of the possible 729 possible such vectors, you can only represent 256 different ones. You have to do more aggressive rounding, but at least the scheme is very simple to decompress and stream.

Woah. Good science.

Thank you for sharing this. I like to test out running LLM's on edge computing with limited RAM and GPU/CPU so this research will have practical implications on my activities. I also appreciated how the authors formulated 1.58 (it's log_2(3)) because that was embarrassingly confusing for me when I was first introduced to ternary LLM's.

[dead]

[flagged]