Think how a reverse polish notation calculator with reasonably limited data stack depth could run efficiently with a plain vanilla transformer.

But if you input classic grade school parenthesized infix with a few levels of operator precedence, you are no longer able to just evaluate the expression during transformer prefill. Even if you add a stack depth bound worth it reasoning tokens between any two input tokens as they're processed.

UTs can, at least if run with encoder (unmasked) attention, resolve the task through technically-flexible iteration count that can and will follow the evaluation order of the infix operator tokens of the input expression.

While masked attention unfortunately limits it's powers, the fundamental benefit of separating task-specific-intelligence (an individual expert of an MoE) from the notion of which transformer layer has it pre-digested just right for that task/processing to be done to it, allows for massive reduction in model parameter count. Note this comes at a penalty of parameter activations (inference will take more compute).

It's just that at some point you can't afford to just train more parameters, without suffering overfitting issues/failures-to-generalize.

The architecture decoupling learned weights from when they're activated also helps with generalization to out-of-distribution structures. Think resilience against yoda-speak and such.