MoE5 articles

MoE

Articles

Tencent Open-Sources Hy4-Preview: 770B MoE Architecture with 1M Context
Tencent has released the open-weights preview of its next-generation foundation model, Hy4-preview, under the Apache 2.0 license. The release scales the organization's Mixture-of-Experts (MoE) line to 770 billion total parameters, activating 49 billion parameters per token across a native 1-million-token context window. Weights have been published across Hugging Face, ModelScope, GitCode, and GitHub, alongside managed API availability on Tencent Cloud TokenHub and OpenRouter. Architectural Sp
August 28, 20261 min
Mixture of Experts: How Sparse Activations Scale Models to Trillions of Parameters Without Trillion-Dollar Bills
Mixture of Experts: How Sparse Activations Scale Models to Trillions of Parameters Without Trillion-Dollar Bills In a dense transformer, every parameter participates in every forward pass. The feed-forward layer — a two-layer perceptron with a hidden expansion of four to eight times the model dimension — alone accounts for roughly two thirds of the FLOPs per token. Scale the model, and cost scales linearly: 10 times the parameters means roughly 10 times the compute at inference, 10 times the me
August 26, 20261 min
Alibaba Releases Open-Weight Qwen3.8-Flash-Next with 6B Active Parameters and Qwen 4 Architecture Preview
Alibaba has released the open weights for Qwen3.8-Flash-Next, a 125B-parameter multimodal Mixture-of-Experts (MoE) model that acts as an early architectural preview for the upcoming Qwen4 family. Operating with only 6B active parameters per token alongside a 51B N-gram embedding layer, the model targets cost efficiency across long-context reasoning, agentic coding, and multimodal workloads. Weights are publicly available on Hugging Face and ModelScope, while a managed production endpoint named
August 26, 20261 min
Serving Mixture-of-Experts Models in Production: Architecture, Distributed Parallelism, All-to-All Bottlenecks, and Serving Economics
Serving Mixture-of-Experts Models in Production: Architecture, Distributed Parallelism, All-to-All Bottlenecks, and Serving Economics Mixture-of-Experts (MoE) architectures have become the standard structural paradigm for frontier open-weight and proprietary large language models. Architectures such as DeepSeek-V3, Mixtral 8x22B, and GLM-5 deliver frontier-grade reasoning and generation by activating only a fraction of their total parameter count on any given token. For example, DeepSeek-V3 rou
August 24, 20261 min
How Mixture of Experts Works: Sparse Gating, Token Routing, and Load Balancing
Large language models face a fundamental scaling trade-off: as parameter counts grow to absorb more world knowledge, the floating-point operations (FLOPs) required per token increase proportionally. In a standard dense Transformer, every single parameter participates in processing every incoming token. Mixture of Experts (MoE) decouples total model capacity from per-token compute cost. By replacing dense feed-forward layers with a dynamic pool of specialized sub-networks ("experts") and routing
August 17, 20261 min

MoE

Articles

Tencent Open-Sources Hy4-Preview: 770B MoE Architecture with 1M Context

Mixture of Experts: How Sparse Activations Scale Models to Trillions of Parameters Without Trillion-Dollar Bills

Alibaba Releases Open-Weight Qwen3.8-Flash-Next with 6B Active Parameters and Qwen 4 Architecture Preview

Serving Mixture-of-Experts Models in Production: Architecture, Distributed Parallelism, All-to-All Bottlenecks, and Serving Economics

How Mixture of Experts Works: Sparse Gating, Token Routing, and Load Balancing