llms.blog
MoE5 articles

MoE

Articles

  • Tencent Open-Sources Hy4-Preview: 770B MoE Architecture with 1M Context

    Tencent has released the open-weights preview of its next-generation foundation model, Hy4-preview, under the Apache 2.0 license. The release scales the organization's Mixture-of-Experts (MoE) line to 770 billion total parameters, activating 49 billion parameters per token across a native 1-million-token context window. Weights have been published across Hugging Face, ModelScope, GitCode, and GitHub, alongside managed API availability on Tencent Cloud TokenHub and OpenRouter. Architectural Sp

    1 min

  • Mixture of Experts: How Sparse Activations Scale Models to Trillions of Parameters Without Trillion-Dollar Bills

    Mixture of Experts: How Sparse Activations Scale Models to Trillions of Parameters Without Trillion-Dollar Bills In a dense transformer, every parameter participates in every forward pass. The feed-forward layer — a two-layer perceptron with a hidden expansion of four to eight times the model dimension — alone accounts for roughly two thirds of the FLOPs per token. Scale the model, and cost scales linearly: 10 times the parameters means roughly 10 times the compute at inference, 10 times the me

    1 min

  • Alibaba Releases Open-Weight Qwen3.8-Flash-Next with 6B Active Parameters and Qwen 4 Architecture Preview

    Alibaba has released the open weights for Qwen3.8-Flash-Next, a 125B-parameter multimodal Mixture-of-Experts (MoE) model that acts as an early architectural preview for the upcoming Qwen4 family. Operating with only 6B active parameters per token alongside a 51B N-gram embedding layer, the model targets cost efficiency across long-context reasoning, agentic coding, and multimodal workloads. Weights are publicly available on Hugging Face and ModelScope, while a managed production endpoint named

    1 min

  • Serving Mixture-of-Experts Models in Production: Architecture, Distributed Parallelism, All-to-All Bottlenecks, and Serving Economics

    Serving Mixture-of-Experts Models in Production: Architecture, Distributed Parallelism, All-to-All Bottlenecks, and Serving Economics Mixture-of-Experts (MoE) architectures have become the standard structural paradigm for frontier open-weight and proprietary large language models. Architectures such as DeepSeek-V3, Mixtral 8x22B, and GLM-5 deliver frontier-grade reasoning and generation by activating only a fraction of their total parameter count on any given token. For example, DeepSeek-V3 rou

    1 min

  • How Mixture of Experts Works: Sparse Gating, Token Routing, and Load Balancing

    Large language models face a fundamental scaling trade-off: as parameter counts grow to absorb more world knowledge, the floating-point operations (FLOPs) required per token increase proportionally. In a standard dense Transformer, every single parameter participates in processing every incoming token. Mixture of Experts (MoE) decouples total model capacity from per-token compute cost. By replacing dense feed-forward layers with a dynamic pool of specialized sub-networks ("experts") and routing

    1 min