Mixture-of-Depths: How Dynamic Compute Allocation and Layer Skipping Scale LLM Efficiency
Standard transformer architectures allocate a uniform computational budget to every token in a sequence. Regardless of whether a model is processing a predictable punctuation mark, a common grammatical connective, or a mathematically dense reasoning step, every token undergoes an identical sequence of matrix multiplications across every multi-head attention and multilayer perceptron (MLP) block throughout the network's depth. This static compute distribution is computationally inefficient. Whil
1 min