Ant Group's AI research lab, inclusionAI, has publicly released six open-weight base checkpoints for its Ling-3.0 foundation model family under the permissive MIT license. The release spans two distinct parameter scales (Ling-3.0-flash and Ling-3.0-tiny) and captures three sequential stages of the pre-alignment training pipeline.
Rather than providing solely post-trained chat models, the release provides unaligned base weights designed specifically for continued pre-training, domain-specific mid-training, fine-tuning, and reinforcement learning research.
Checkpoints Across Three Training Stages
The repository structure covers two model sizes across three distinct phases of the pre-training lifecycle:
- Pre-trained Checkpoints:
Ling-3.0-flash-base-30TandLing-3.0-tiny-base-30Treflect raw models immediately following large-scale 30-trillion-token pre-training, prior to mid-training or checkpoint merging. - Mid-trained Checkpoints:
Ling-3.0-flash-base-midtrainandLing-3.0-tiny-base-midtraincapture weights after mid-training data phases. - Merged Base Checkpoints:
Ling-3.0-flash-baseandLing-3.0-tiny-baserepresent the final base state produced by Warmup-Stable and Merge (WSM) before post-training.
All six checkpoints are strictly base models without supervised fine-tuning, preference optimization, or safety alignment.
The Warmup-Stable and Merge (WSM) Approach
A core architectural characteristic of the release is the replacement of conventional learning rate decay schedules with Warmup-Stable and Merge (WSM). Under standard training setups, cosine or linear learning rate decay at the end of pre-training bakes the final data distribution into model weights, making subsequent continued pre-training prone to instability or catastrophic forgetting.
Under WSM, the model trains at a stable learning rate through the end of the run. Intermediate checkpoints captured along the stable trajectory are subsequently combined using learned merge weights. This structure allows teams to:
- Perform continued pre-training and domain-specific dataset expansion without fighting an decayed learning rate.
- Explore alternative merge profiles offline without incurring the compute costs of full retraining runs.
- Maintain shared training recipes between
Ling-3.0-tiny(7.9B total, 1.3B active) andLing-3.0-flash(124B total, 5.1B active), enabling rapid low-cost validation on small hardware before scaling to 124B parameters.
Architectural Specifications
Ling-3.0-flash-base uses the BailingMoeV3ForCausalLM sparse mixture-of-experts architecture:
- Parameter Count: 124 billion non-embedding parameters (approximately 127.5 billion total parameters on disk including embedding layers), with 5.1 billion parameters activated per token.
- MoE Routing: A 1/64 sparsity pattern across 512 routed experts, activating 8 routed experts and 1 shared expert per token.
- Hybrid Attention: 42 transformer layers structured in a 5:1 repeating pattern, combining 35 Kimi Delta Attention (KDA) linear attention layers with 7 Gated Multi-Head Latent Attention (MLA) layers.
- Context Length: Native 256K context window (262,144 tokens).
- Multi-Token Prediction: A 3.1-billion-parameter Multi-Token Prediction (MTP) module is included for speculative decoding workflows.
- Licensing: Released under the MIT license without proprietary commercial restrictions.
The checkpoints are available on Hugging Face in BF16 format with inference and serving support validated in vLLM and SGLang.
