Subscribe
Topic

Mixture of Experts

Mixture of Experts (MoE) models route each token to a few expert networks. Posts explain MoE routing and load balancing, including auxiliary-loss-free balancing and expert parallelism, and cover open MoE releases such as Reflection AI's Beam 1.0.

Articles

1-4
  1. Reflection AI Releases Beam 1.0: 501B Open-Weight MoE Under Apache 2.0

    US AI research lab Reflection AI has released Beam 1.0, a 501-billion-parameter open-weight Mixture-of-Experts (MoE) foundation model licensed under Apache 2.0. The release includes weights in BF16, FP8, and NVFP4 precision formats alongside training harnesses and inference integration for vLLM, SGLang, and Ollama. Architecture and Sparse Compute Profile Beam is built across 52 layers with an interleaved local and global attention structure, maintaining a 3:1 ratio between sliding-window and

    Open Source
    2 min read
  2. Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations of Noisy Top-k Gating, Expert Capacity, Auxiliary Balancing Losses, and Loss-Free Load Balancing

    Mixture-of-Experts (MoE) Routing and Load Balancing: Mathematical Foundations of Noisy Top-k Gating, Expert Capacity, Auxiliary Balancing Losses, and Loss-Free Load Balancing Scaling dense transformer models incurs computational costs that grow proportionally with parameter count. In dense architectures, every token in a sequence activates all feed-forward network (FFN) parameters across every layer. Sparse Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute b

    AI
    15 min read
  3. Auxiliary-Loss-Free Load Balancing in Mixture-of-Experts: How Dynamic Bias Adjustments Eliminate Gradient Conflict and Routing Collapse

    Sparse Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute cost by activating only a small subset of feed-forward network (FFN) parameters for any given token. While dense transformers evaluate every parameter across all sequence positions, MoE models route tokens dynamically to specialized sub-networks, enabling parameter scaling to hundreds of billions or trillions of parameters at the inference and training cost of much smaller dense models. However, condit

    AI
    9 min read
  4. Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures

    Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures Scaling dense Large Language Models (LLMs) requires activating every parameter in the network for every token in a sequence. While techniques like Tensor Parallelism, Pipeline Parallelism, and Fully Sharded Data Parallelism distribute billions of dense parameters across clusters, computational cost scales linearly with parameter count. Mixture-of-Experts (Mo

    AI
    9 min read