Continuous Pre-Training in Production: Domain Adaptation, Replay Buffers, Learning Rate Restarts, and Catastrophic Forgetting Mitigation

Adapting general-purpose foundation models to specialized enterprise domains (such as clinical medicine, corporate law, quantitative finance, and proprietary software codebases) presents a fundamental architectural challenge. While Retrieval-Augmented Generation (RAG) and Supervised Fine-Tuning (SFT) remain standard first-line approaches, both exhibit severe structural limitations when models require deep, systemic domain competence. SFT primarily modifies conversational style, output schema compliance, and task formatting rather than injecting dense factual knowledge graphs into parametric memory. RAG introduces context-window latency, token cost overheads, and retrieval failure modes when queries require multi-document synthesis over intricate domain ontologies.

Continuous Pre-Training (CPT), also known as continual pre-training or second-stage pre-training, bridges this gap by continuing causal language modeling on an existing pre-trained checkpoint using billions of domain-specific tokens. However, executing CPT in production environments introduces the stability-plasticity dilemma: networks rapidly suffer from catastrophic forgetting, degrading general reasoning, mathematics, and instruction-following performance while assimilating new domain representations. Managing CPT requires careful coordination of tokenizer adaptation, replay buffer composition, learning rate re-warming schedules, and distributed training infrastructure.

The Stability-Plasticity Trade-Off in Domain Adaptation

When an autoregressive transformer trained on trillions of tokens from broad web corpora is exposed exclusively to a narrow domain corpus, gradient descent aggressively overwrites orthogonal weight representations. Research documented by Gupta et al. (2023) and Ibrahim et al. (2024) demonstrates that standard cross-entropy loss optimization on domain-only data causes rapid performance collapse on out-of-domain benchmarks such as MMLU, GSM8K, and HumanEval within a fraction of an epoch.

+-----------------------------------------------------------------------------+
|               CONTINUOUS PRE-TRAINING DATA MIXING & REPLAY PIPELINE         |
+-----------------------------------------------------------------------------+
|                                                                             |
|   +--------------------------+          +-------------------------------+   |
|   | Domain Corpus (80-90%)   |          | General Replay Buffer (10-20%)|   |
|   | - Technical papers       |          | - FineWeb / SlimPajama subset |   |
|   | - Internal codebases     |          | - Mathematical reasoning      |   |
|   | - Regulatory filings     |          | - Instruction demonstrations  |   |
|   +------------+-------------+          +---------------+---------------+   |
|                |                                        |                   |
|                +-------------------+--------------------+                   |
|                                    |                                        |
|                                    v                                        |
|                    +-------------------------------+                        |
|                    | Deterministic Batch Sampler   |                        |
|                    | (Interleaved Token Packing)   |                        |
|                    +---------------+---------------+                        |
|                                    |                                        |
|                                    v                                        |
|                    +-------------------------------+                        |
|                    | Transformer Forward / Backward|                        |
|                    | (Distributed FSDP / Megatron) |                        |
|                    +---------------+---------------+                        |
|                                    |                                        |
|                                    v                                        |
|                    +-------------------------------+                        |
|                    | Re-warmed LR Optimizer Step   |                        |
|                    | (10-20% Peak LR, WSD/Cosine)  |                        |
|                    +-------------------------------+                        |
+-----------------------------------------------------------------------------+

The core failure mechanisms during unconstrained continual training include:

1. Representation Drift: Feature extractors in early and middle transformer layers adapt their attention projections to domain-specific syntax, destabilizing downstream head activations.
2. Gradient Interference: Domain gradients conflict directly with orthogonal sub-networks responsible for multi-step logic and general language modeling.
3. Loss of Regularization Signals: Web pre-training exposes models to diverse syntactical structures; highly repetitive domain data leads to sharp local minima and loss spikes.

Tokenizer Adaptation: Vocabulary Expansion vs. Fixed Tokenizers

A primary design choice in continuous pre-training is whether to expand the base model's byte-pair encoding (BPE) tokenizer to include specialized domain vocabulary.

Keeping Fixed Tokenizers

Modern foundation models (such as Llama 3 with a 128,000-token vocabulary or Qwen 2.5 with 151,000+ tokens) already allocate substantial subword capacity to technical terms. Maintaining a fixed vocabulary eliminates the need to resize embedding matrices or language model heads. This preserves the geometric alignment of existing token embeddings and prevents early training instability. The trade-off is higher sequence fertility: specialized chemical compounds, medical terminology, or proprietary API calls decompose into multiple subword tokens, consuming more sequence context and compute per document.

Expanding Domain Vocabulary

Expanding the tokenizer by adding 5,000 to 20,000 domain-specific tokens reduces sequence length on domain datasets by 20% to 35%, cutting prefill compute and training time. However, new vocabulary entries introduce uninitialized rows in the input embedding tensor $W_e \in \mathbb{R}^{V \times d}$ and the output lm_head $W_{out} \in \mathbb{R}^{V \times d}$.

To avoid destructive loss spikes during initial optimization steps, teams utilize specific initialization heuristics:

• Subword Averaging: Initializing new token vectors by computing the mean embedding of the constituent subwords that previously represented the term.
• Neighborhood Projection: Projecting new token vectors using nearest-neighbor representations computed from a domain-specific continuous bag-of-words (CBOW) or Word2Vec embedding space.
• Selective Embedding Warming: Freezing transformer backbone layers for the first 500 to 1,000 steps while training only the newly added embedding rows to reach numerical equilibrium with the pre-trained latent space.

Catastrophic Forgetting Mitigation: Replay Buffers and Synthetic Data

Empirical studies from Cossu et al. (2022) and Ibrahim et al. (2024) confirm that data replay is the single most reliable mechanism for mitigating catastrophic forgetting in autoregressive language models.

Replay Buffer Sizing and Composition

Rather than training exclusively on the domain corpus, production CPT pipelines interleave a fixed percentage of general pre-training data into every training batch:

• Replay Ratio: A general replay mix of 10% to 20% (drawn from high-quality web datasets such as FineWeb, SlimPajama, or synthetic reasoning corpora) preserves 95% to 98% of baseline benchmark performance (MMLU, GSM8K, ARC-Challenge) with minimal degradation to domain absorption rates.
• Domain Replay vs. General Replay: If the foundation model undergoes multi-stage CPT across consecutive domains (e.g., General -> Legal -> Tax Law), the replay buffer must include samples from both the base foundation distribution and the prior domain to prevent sequential degradation.
• Batch-Level Interleaving: Packing general and domain sequences into the same micro-batch yields better gradient stability than alternating pure domain batches and pure replay batches.

Reading Comprehension and Synthetic Augmentation

Raw unstructured domain text (such as raw medical manuals or corporate wiki dumps) often yields weak gradient signals during standard next-token prediction. Following the AdaptLLM methodology introduced by Cheng et al. (2023), transforming raw domain text into reading comprehension exercises, structured Q&A pairs, and summary tasks significantly improves token efficiency. This synthetic structuring encourages the model to learn bidirectional factual relationships and factual retrieval rather than merely memorizing local phrase co-occurrences.

Optimizer Dynamics and Learning Rate Schedules

Foundation models typically conclude their original pre-training run at a decayed minimum learning rate (often $\eta_{min} \approx 10^{-6}$ or $0.1 \times \eta_{peak}$). Resuming training from this state requires a deliberate learning rate policy.

Learning
Rate (LR)
  ^
  |        Pre-Training           Re-warming       Domain Decay
  |        Decay Phase               Phase             Phase
  |     \                         /----\
  |      \                       /      \
  |       \                     /        \
  |        \                   /          \
  |         \                 /            \
  |          \               /              \
  |           \-------------/                \---------------
  +------------------------------------------------------------> Training Steps
      Foundation Pre-Training       Continuous Pre-Training (CPT)

Re-Warming and Re-Decaying

Continuing pre-training at $\eta_{min}$ leads to severe underfitting; the model fails to acquire new domain knowledge within reasonable compute budgets. Conversely, restarting training at the original base model peak learning rate ($\eta_{peak} \approx 3 \times 10^{-4}$) shatters pre-trained feature weights.

As established by Gupta et al. (2023), the optimal strategy is a re-warming and re-decaying schedule:

1. Peak Learning Rate Calibration: Set the CPT peak learning rate $\eta_{CPT}$ to between 10% and 25% of the foundation model's original peak learning rate (typically $3 \times 10^{-5}$ to $1 \times 10^{-4}$ for standard 7B to 70B parameter models).
2. Warmup Duration: Apply a linear or cosine warmup over 1% to 5% of the total allocated CPT token budget to allow optimizer moment statistics to adjust to domain gradients.
3. Decay Trajectory: Follow a cosine or linear decay down to $0.1 \times \eta_{CPT}$.

Warmup-Stable-Decay (WSD) in Continual Learning

When the total token volume of the domain corpus is uncertain or expanding continuously, standard cosine schedules are problematic because the decay trajectory must be parameterized against a fixed final step count $T$.

The Warmup-Stable-Decay (WSD) schedule resolves this operational friction:

• The learning rate warms up to $\eta_{CPT}$ and remains flat during a prolonged stable phase.
• Checkpoints can be branched or evaluated continuously during the stable phase.
• When a production release is scheduled, a short cooldown phase (typically over the final 10% to 15% of tokens) decays the learning rate to zero, solidifying parametric consolidation.

Optimizer State Reset vs. Preservation

When initializing CPT from an open-weight release, pre-training optimizer states ($m_t, v_t$ in AdamW) are rarely published. Initializing AdamW with clean optimizer states causes initial momentum mismatches. Employing a short warmup phase (500 to 2,000 steps) with decoupled weight decay ($0.01$ to $0.1$) allows the second moment buffer $v_t$ to accurately estimate gradient variance without inducing weight instability.

Production Frameworks and Compute Economics

Continuous pre-training typically operates on corpora ranging from 5 billion to 100 billion tokens. For an 8B-parameter model, training on 20 billion tokens requires approximately 960 H100-hours (assuming standard Model Flops Utilization of 45% to 50%).

Comparing Adaptation Paradigms

• Supervised Fine-Tuning (SFT): Operates on 10M to 500M tokens (1 to 20 GPU-hours for an 8B model). Focuses on conversational task alignment, schema compliance, and tool calling. Does not inject dense factual graphs; prone to hallucination when queried on unfamiliar domain entities.
• Continuous Pre-Training (CPT): Operates on 5B to 100B tokens (250 to 5,000 GPU-hours for an 8B model). Injects deep domain vocabulary, technical literature, and specialized syntactic patterns. Requires distributed training, data replay buffers, and calibrated optimizer schedules.
• Pre-Training from Scratch: Operates on 2T to 15T+ tokens (100,000+ GPU-hours). Establishes broad baseline world knowledge and multi-task reasoning. Carries prohibitive capital expenditure for single-domain enterprise adaptations.

Distributed Training Runtimes

• PyTorch FSDP / Torchtune: Suitable for single-node to medium-cluster (8 to 64 GPUs) CPT runs. FSDP2 with per-parameter sharding minimizes communication overhead while supporting gradient checkpointing and mixed-precision FP8/BF16 execution.
• Megatron-LM / Nanotron: Optimized for large-scale multi-node deployments (128+ GPUs) requiring 3D parallelism (Tensor, Pipeline, and Data Parallelism) with sequence packing and asynchronous checkpoint staging.

Engineering Checklist for Continuous Pre-Training

1. Audit Domain Data Quality: Deduplicate text with MinHash LSH; strip low-signal formatting; filter out machine-generated slop; verify license and data lineage.
2. Determine Tokenizer Strategy: If domain sequence compression exceeds 25% with new tokens, expand vocabulary and initialize embeddings via subword averaging; otherwise, lock the base tokenizer.
3. Establish a Replay Stream: Reserve 10% to 20% of every training batch for general pre-training data (e.g., FineWeb/SlimPajama).
4. Configure LR and Optimizer: Set peak learning rate to 10-20% of original pre-training peak; implement a 2-5% linear warmup; use WSD or cosine decay.
5. Set Up Real-Time Validation Probes: Run continuous evaluations across domain validation perplexity, general reasoning benchmarks (MMLU, GSM8K), and downstream task probes at regular checkpoint intervals.

Sources

• Simple and Scalable Strategies to Continually Pre-train Large Language Models (arXiv:2403.08763)
• Continual Pre-Training of Large Language Models: How to (re)warm your model? (arXiv:2308.04014)
• Adapting Large Language Models via Reading Comprehension (arXiv:2309.09530)
• Towards Effective and Efficient Continual Pre-training of Large Language Models (arXiv:2407.18743)
• Continual Pre-training of Language Models (arXiv:2302.03241)
• Learning Dynamics in Continual Pre-Training for Large Language Models (arXiv:2505.07796)