inclusionAI Releases Six Ling-3.0 Base Checkpoints with Warmup-Stable-and-Merge Architecture

Ant Group's AI research lab, inclusionAI, has publicly released six open-weight base checkpoints for its Ling-3.0 foundation model family under the permissive MIT license. The release spans two distinct parameter scales (Ling-3.0-flash and Ling-3.0-tiny) and captures three sequential stages of the pre-alignment training pipeline.

Rather than providing solely post-trained chat models, the release provides unaligned base weights designed specifically for continued pre-training, domain-specific mid-training, fine-tuning, and reinforcement learning research.

Checkpoints Across Three Training Stages

The repository structure covers two model sizes across three distinct phases of the pre-training lifecycle:

• Pre-trained Checkpoints: Ling-3.0-flash-base-30T and Ling-3.0-tiny-base-30T reflect raw models immediately following large-scale 30-trillion-token pre-training, prior to mid-training or checkpoint merging.
• Mid-trained Checkpoints: Ling-3.0-flash-base-midtrain and Ling-3.0-tiny-base-midtrain capture weights after mid-training data phases.
• Merged Base Checkpoints: Ling-3.0-flash-base and Ling-3.0-tiny-base represent the final base state produced by Warmup-Stable and Merge (WSM) before post-training.

All six checkpoints are strictly base models without supervised fine-tuning, preference optimization, or safety alignment.

The Warmup-Stable and Merge (WSM) Approach

A core architectural characteristic of the release is the replacement of conventional learning rate decay schedules with Warmup-Stable and Merge (WSM). Under standard training setups, cosine or linear learning rate decay at the end of pre-training bakes the final data distribution into model weights, making subsequent continued pre-training prone to instability or catastrophic forgetting.

Under WSM, the model trains at a stable learning rate through the end of the run. Intermediate checkpoints captured along the stable trajectory are subsequently combined using learned merge weights. This structure allows teams to:

• Perform continued pre-training and domain-specific dataset expansion without fighting an decayed learning rate.
• Explore alternative merge profiles offline without incurring the compute costs of full retraining runs.
• Maintain shared training recipes between Ling-3.0-tiny (7.9B total, 1.3B active) and Ling-3.0-flash (124B total, 5.1B active), enabling rapid low-cost validation on small hardware before scaling to 124B parameters.

Architectural Specifications

Ling-3.0-flash-base uses the BailingMoeV3ForCausalLM sparse mixture-of-experts architecture:

• Parameter Count: 124 billion non-embedding parameters (approximately 127.5 billion total parameters on disk including embedding layers), with 5.1 billion parameters activated per token.
• MoE Routing: A 1/64 sparsity pattern across 512 routed experts, activating 8 routed experts and 1 shared expert per token.
• Hybrid Attention: 42 transformer layers structured in a 5:1 repeating pattern, combining 35 Kimi Delta Attention (KDA) linear attention layers with 7 Gated Multi-Head Latent Attention (MLA) layers.
• Context Length: Native 256K context window (262,144 tokens).
• Multi-Token Prediction: A 3.1-billion-parameter Multi-Token Prediction (MTP) module is included for speculative decoding workflows.
• Licensing: Released under the MIT license without proprietary commercial restrictions.

The checkpoints are available on Hugging Face in BF16 format with inference and serving support validated in vLLM and SGLang.

Sources

• inclusionAI Ling-3.0-flash on Hugging Face
• inclusionAI Ling-3.0-tiny on Hugging Face
• OrcaRouter: Ling-3.0-flash-base Open Checkpoint Analysis