I get the impression their setup is very hard to maintain but it's worth every penny. They've done optimizations that wring incredible performance out of the hardware they have, but they also have specific machine configurations and I wouldn't be surprised if they have complicated hacks that get 100% speedups for some stuff but those speedups disappear if you have a slightly different motherboard configuration. Also there's suggestion they've made firmware hacks which are worth it at their scale, but might be very dangerous and difficult to apply especially on a small scale. (And some of their hacks might involve both firmware and cluster-level optimizations, which would be useless or counterproductive independently.)

And even if you have somewhat similar hardware, the code might not be that helpful, you might be better off with a sketch of the solution and implementing it yourself. If you've got a large enough cluster it's going to pay for itself anyway.

Unmaintainable seems unduly harsh. There is a big gap between maintainable internally and ready for public consumption

They're going to spend time and effort into making their optimizations public. Would you rather have them keep their changes internal?

There's an ongoing debate whether LLM should be considered intelligent when it's just generating tokens from latent space. Meanwhile there are humans that are only capable of spitting out the same 5 tokens yet still considered to be "intelligent".