hey thanks for the comment, author here. That's a fair read of the loss numbers, I could only fund up to 1.2B scale myself and I don't want to oversell the results. But note that the benefits that Soft-MTP delivers in the paper is *consistently growing across scale*. The residual is irreducible with scale and will only become noisier as model grows bigger, so removing it via Soft-MTP is intuitive. It is also easily verifiable cos it only needs minor code change, codebase:
https://github.com/JaydenTeoh/Soft-MTPthere is reproduction of Soft-MTP's benefit over standard MTP at 35B scale but it is only one data point for now: https://x.com/HivazeMe/status/2118432486724391217