Mixture of Experts4 articles

Mixture of Experts

Articles

Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations, Top-k Gating, Auxiliary Losses, Auxiliary-Loss-Free Balancing, and Distributed Expert Parallelism
Scaling autoregressive language models under fixed inference latency and training compute budgets has forced a departure from purely dense feed-forward architectures. In a standard dense transformer, every parameter is activated for every token, resulting in computational complexity and memory bandwidth consumption that scale linearly with the total parameter count. Mixture-of-Experts (MoE) architectures decouple parameter capacity from per-token compute by replacing monolithic Multi-Layer Perc
August 28, 20261 min
Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations of Noisy Top-k Gating, Expert Capacity, Auxiliary Balancing Losses, and Loss-Free Load Balancing
Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations of Noisy Top-k Gating, Expert Capacity, Auxiliary Balancing Losses, and Loss-Free Load Balancing Scaling dense transformer models incurs computational costs that grow proportionally with parameter count. In dense architectures, every token in a sequence activates all feed-forward network (FFN) parameters across every layer. Sparse Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute b
August 27, 20261 min
Auxiliary-Loss-Free Load Balancing in Mixture-of-Experts: How Dynamic Bias Adjustments Eliminate Gradient Conflict and Routing Collapse
Sparse Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute cost by activating only a small subset of feed-forward network (FFN) parameters for any given token. While dense transformers evaluate every parameter across all sequence positions, MoE models route tokens dynamically to specialized sub-networks, enabling parameter scaling to hundreds of billions or trillions of parameters at the inference and training cost of much smaller dense models. However, condit
August 21, 20261 min
Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures
Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures Scaling dense Large Language Models (LLMs) requires activating every parameter in the network for every token in a sequence. While techniques like Tensor Parallelism, Pipeline Parallelism, and Fully Sharded Data Parallelism distribute billions of dense parameters across clusters, computational cost scales linearly with parameter count. Mixture-of-Experts (Mo
August 19, 20261 min

Mixture of Experts

Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations, Top-k Gating, Auxiliary Losses, Auxiliary-Loss-Free Balancing, and Distributed Expert Parallelism

Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations of Noisy Top-k Gating, Expert Capacity, Auxiliary Balancing Losses, and Loss-Free Load Balancing

Auxiliary-Loss-Free Load Balancing in Mixture-of-Experts: How Dynamic Bias Adjustments Eliminate Gradient Conflict and Routing Collapse

Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures