Subscribe
Topic

MoE

Coverage of MoE (Mixture of Experts) models and how they work. Posts explain sparse gating, token routing and load balancing, cover serving MoE models in production, and track MoE releases like Tencent's Hy4-Preview and Qwen3.8-Flash-Next.

Articles

1-5
  1. Tencent Open-Sources Hy4-Preview: 770B MoE Architecture with 1M Context

    Tencent has released the open-weights preview of its next-generation foundation model, Hy4-preview, under the Apache 2.0 license. The release scales the organization's Mixture-of-Experts (MoE) line to 770 billion total parameters, activating 49 billion parameters per token across a native 1-million-token context window. Weights have been published across Hugging Face, ModelScope, GitCode, and GitHub, alongside managed API availability on Tencent Cloud TokenHub and OpenRouter. Update, October 2

    AI
    2 min read
  2. Mixture of Experts: How Sparse Activations Scale Models to Trillions of Parameters Without Trillion-Dollar Bills

    Mixture of Experts: How Sparse Activations Scale Models to Trillions of Parameters Without Trillion-Dollar Bills In a dense transformer, every parameter participates in every forward pass. The feed-forward layer — a two-layer perceptron with a hidden expansion of four to eight times the model dimension — alone accounts for roughly two thirds of the FLOPs per token. Scale the model, and cost scales linearly: 10 times the parameters means roughly 10 times the compute at inference, 10 times the me

    AI
    7 min read
  3. Alibaba Releases Open-Weight Qwen3.8-Flash-Next with 6B Active Parameters and Qwen 4 Architecture Preview

    Alibaba has released the open weights for Qwen3.8-Flash-Next, a 125B-parameter multimodal Mixture-of-Experts (MoE) model that acts as an early architectural preview for the upcoming Qwen4 family. Operating with only 6B active parameters per token alongside a 51B N-gram embedding layer, the model targets cost efficiency across long-context reasoning, agentic coding, and multimodal workloads. For how it fits with Qwen3.8-Max, the open 2.4T weights and the Omni models, see our Qwen3.8 explainer. U

    Alibaba
    2 min read
  4. Serving Mixture-of-Experts Models in Production: Architecture, Distributed Parallelism, All-to-All Bottlenecks, and Serving Economics

    Serving Mixture-of-Experts Models in Production: Architecture, Distributed Parallelism, All-to-All Bottlenecks, and Serving Economics Mixture-of-Experts (MoE) architectures have become the standard structural paradigm for frontier open-weight and proprietary large language models. Architectures such as DeepSeek-V3, Mixtral 8x22B, and GLM-5 deliver frontier-grade reasoning and generation by activating only a fraction of their total parameter count on any given token. For example, DeepSeek-V3 rou

    AI
    8 min read
  5. How Mixture of Experts Works: Sparse Gating, Token Routing, and Load Balancing

    Large language models face a fundamental scaling trade-off: as parameter counts grow to absorb more world knowledge, the floating-point operations (FLOPs) required per token increase proportionally. In a standard dense Transformer, every single parameter participates in processing every incoming token. Mixture of Experts (MoE) decouples total model capacity from per-token compute cost. By replacing dense feed-forward layers with a dynamic pool of specialized sub-networks ("experts") and routing

    AI
    6 min read