Subscribe
5 min read

What Is a Mixture-of-Experts Model? Why AI Giants Only Wake Part of Themselves

Total parameters sit in memory. Active parameters cost compute. The arithmetic behind Mixtral, DeepSeek-V3, Kimi K2 and the gpt-oss models, and why the model names no longer tell you what you pay for.

Olivetti-style poster: WHAT IS A MIXTURE OF EXPERTS? with a selector arm directing one token into two of a row of boxy expert machines
Olivetti-style poster: WHAT IS A MIXTURE OF EXPERTS? with a selector arm directing one token into two of a row of boxy expert machines

When OpenAI released gpt-oss-120b in August 2025, its first open-weight models since 2019, the "120b" in the name was doing less work than it appears. The model holds about 117 billion parameters in total, but only 5.1 billion of them activate for any given token. It fits on a single 80GB Nvidia A100 because, on any forward pass, it behaves roughly like a 5-billion-parameter network.

The trick is called a mixture-of-experts (MoE) architecture, and it now sits under a growing share of the frontier models people actually use: Mixtral, DeepSeek-V3, Kimi K2, Qwen3, Llama 4, Grok-1 and the gpt-oss line are all MoE models.

A rotary selector dial routing signals to a bank of expert units, two lit

The basic swap

A transformer block has two main parts: an attention layer that lets tokens look at each other, and a feedforward network (FFN) that processes each token independently. In a dense model, every token passes through one FFN per layer, and that FFN is always the same set of weights.

An MoE model replaces some of those FFNs with a row of parallel FFNs, called experts, plus a small gating network called a router. For each token at each MoE layer, the router scores every expert and sends the token to the top-k of them, typically two, four or eight out of dozens or hundreds. Only the selected experts do any computation. The attention layers, embeddings and layer norms stay shared: every token uses them.

The modern form comes from a 2017 Google paper, "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer and colleagues, which proposed stacking MoE layers with a learned gate so a network could grow its parameter count enormously while keeping the compute cost per token fixed. Google's GShard system (2020) scaled the idea to a 600-billion-parameter translation model with 2,048 experts, and the Switch Transformer (2021) simplified routing to one expert per layer, reaching 1.6 trillion parameters and reporting roughly seven-times pre-training speedups over a comparably sized dense T5.

Do the arithmetic

Mixtral 8x7B, released by Mistral AI in December 2023 under an Apache 2.0 license, is the cleanest worked example. It has 32 transformer layers, and in each MoE layer there are 8 expert FFNs, of which the router picks 2 per token.

The arithmetic works out to 46.7 billion total parameters, but only 12.9 billion active per token. The "8x7B" name implies 56 billion; the real number is lower because only the FFNs are multiplied, not the attention blocks. Roughly: shared components contribute 1.6 billion, each expert across all layers about 5.6 billion, so a token pays 1.6 + 2 x 5.6 = 12.9 billion.

DeepSeek-V3 (December 2024, 671 billion total, 37 billion active) went further in the opposite direction: 256 small routed experts per layer with 8 selected, plus 1 always-on shared expert. This "fine-grained" recipe, introduced in DeepSeek-V2, means the model composes each token's computation out of many narrow specialists rather than a few wide ones. DeepSeek reported training the whole model on 2.788 million H800 GPU-hours, a number that circulated widely because it was far below what peers were spending on models of similar published size.

Other 2025-era checkpoints: Qwen3's flagship 235B-A22B model runs 22 billion active parameters per token; Moonshot's Kimi K2 holds 1 trillion total with 32 billion active across 384 experts; Meta's Llama 4 Scout and Maverick ship 109 billion and 400 billion total parameters respectively, both activating 17 billion; OpenAI's gpt-oss-120b uses 128 experts per layer with 4 active.

What the router is actually for

The router is a small learned network that turns a token's hidden state into a score per expert. Two problems shaped its design.

First, collapse. If the router learns to favor a few experts, the others get no gradient and never improve, so the model wastes most of its parameters. Papers since Shazeer's have added auxiliary load-balancing losses that penalize uneven routing, and DeepSeek-V3 further uses sequence-level balance loss and routing bias adjustments to keep expert utilization even.

Second, specialization is emergent, not assigned. Nobody labels one expert "the French grammar department." Probing studies on Mixtral-era models found measurable clustering (code-related tokens, punctuation, certain languages tending toward particular experts), but experts overlap heavily and individual experts are not cleanly interpretable units. Training-time randomization in routing, a trick from Shazeer's paper onward, keeps that overlap healthy.

The economic bargain, and its catch

MoE trades memory for compute. All parameters must sit in memory or on fast storage, because any expert can be called at any layer for any token. A 16-bit Mixtral checkpoint needs about 90GB or more of weight storage. But the arithmetic per token, the part that dominates serving cost at scale, corresponds to the 12.9 billion active parameters, not the 46.7 billion total.

A balance scale weighing a full cabinet of drawers against one glowing open drawer

That asymmetry explains a visible slice of the current pricing landscape around Chinese open-weight labs. Inference providers can serve a 671-billion-parameter DeepSeek model at per-token compute closer to a mid-size dense model, which helps explain why its API prices undercut western frontier models by an order of magnitude while posting competitive benchmark scores. The catch shows up at small batch sizes and on consumer hardware: every request may touch a different expert subset, so the whole weight set must stay resident even though each token uses a slice. Dense models, by contrast, let a small GPU hold exactly what it will use.

One more consequence is purely linguistic. A headline parameter count like "1 trillion" (Kimi K2) or "400 billion" (Llama 4 Maverick) now describes capacity on disk, not cost per token. Active parameters are the number that tracks speed and price. When a model card states both, as gpt-oss and Qwen3 do, the ratio between them, 5.1B/117B in OpenAI's case, is the sparsity, and it keeps getting more extreme.

Sources

  • Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (arXiv:1701.06538)
  • Lepikhin et al., GShard (arXiv:2006.16668); Fedus et al., Switch Transformers (arXiv:2101.03961)
  • Mistral AI, Mixtral of Experts (arXiv:2401.04088) and the December 2023 Mixtral announcement
  • DeepSeek-AI, DeepSeek-V2 (arXiv:2405.04434) and DeepSeek-V3 Technical Report (arXiv:2412.19437)
  • OpenAI, Introducing gpt-oss (August 2025), including the model table of total/active parameters and expert counts
  • Qwen Team, Qwen3 Technical Report (arXiv:2505.09388); Kimi K2 open weights release (arXiv:2507.20534)
  • Meta Llama 4 launch materials (April 2025)