Pre-Training9 articles

Pre-Training

Articles

Multi-Token Prediction (MTP): Mathematical Foundations, Shared Trunk Architectures, Sequential Future Verification, and Speculative Decoding Dynamics
The standard training objective for autoregressive large language models is next-token prediction (NTP), where model parameters $\theta$ are trained via maximum likelihood estimation to forecast a single subsequent token given all previous context. While this paradigm has driven modern foundation models, it enforces a myopic local optimization: the model learns transition probabilities strictly between adjacent tokens without explicit incentives to plan multi-step syntactic or semantic trajector
August 29, 20261 min
Data Pruning and Core-Set Selection in Deep Learning: How EL2N, GraNd, and Memorization Dynamics Break Power-Law Scaling
Modern foundation models are trained on tens of trillions of tokens, requiring millions of GPU hours. Yet empirical analyses consistently reveal that massive portions of web-crawled corpora and large-scale vision datasets are either highly redundant, uninformative, or dominated by unlearnable noise. Standard empirical scaling laws (such as those formulated by Kaplan et al. and Chinchilla) model generalization error as a power-law function of total training samples ($L(N) \propto N^{-\alpha}$). H
August 24, 20261 min
Continuous Pre-Training in Production: Domain Adaptation, Replay Buffers, Learning Rate Restarts, and Catastrophic Forgetting Mitigation
Continuous Pre-Training in Production: Domain Adaptation, Replay Buffers, Learning Rate Restarts, and Catastrophic Forgetting Mitigation Adapting general-purpose foundation models to specialized enterprise domains (such as clinical medicine, corporate law, quantitative finance, and proprietary software codebases) presents a fundamental architectural challenge. While Retrieval-Augmented Generation (RAG) and Supervised Fine-Tuning (SFT) remain standard first-line approaches, both exhibit severe s
August 23, 20261 min
Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training
Weight Initialization in Large Language Models: How Variance Scaling, Residual Multipliers, and DeepNorm Stabilize Deep Transformer Pre-Training In deep transformer architectures, weight initialization is the primary determinant of whether a trillion-token pre-training run converges smoothly or diverges during the first thousand steps. When training networks with 80 to 120 layers (such as Llama 3 70B, GPT-4, or deep mixture-of-experts models), naive application of classical Gaussian or uniform
August 21, 20261 min
Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics
Data Mixing and Domain Scheduling in Large Language Models: How DoReMi, RegMix, and Multi-Stage Annealing Shape Pre-Training Dynamics In large language model pre-training, data composition is as consequential as parameter count and compute budget. While early foundation models relied on raw natural frequencies or manual heuristic filtering to construct training corpora, empirical scaling laws have shown that arbitrary domain ratios cause severe compute inefficiencies. Over-sampling redundant te
August 21, 20261 min
Loss Spikes and Training Stability in Large Language Models: How Attention Logit Drift, z-loss, and QK-Norm Prevent Gradient Explosions
During the pre-training of modern large language models, few operational failures are as costly as loss spikes. When training clusters containing thousands of GPUs run for weeks across trillions of tokens, a sudden, discontinuous surge in cross-entropy loss can corrupt optimizer momentum buffers, induce numerical overflow in half-precision representations, and permanently degrade downstream model capabilities. In severe cases, models experience catastrophic divergence, forcing engineering teams
August 21, 20261 min
Adaptive Optimizers in Large Language Model Pre-Training: How AdamW, Adafactor, and Lion Scale Gradient Updates Across Billions of Parameters
Adaptive Optimizers in Large Language Model Pre-Training: How AdamW, Adafactor, and Lion Scale Gradient Updates Across Billions of Parameters Large language model pre-training requires optimizing billions of parameters over trillions of tokens across distributed GPU clusters. Standard stochastic gradient descent (SGD) fails in this regime because Transformer loss landscapes are severely ill-conditioned, with gradient magnitudes differing by orders of magnitude across layers and token positions.
August 20, 20261 min
Data Filtering and Deduplication in Large Language Models: How Heuristics, MinHash LSH, SemDeDup, and Quality Classifiers Curate Pre-Training Corpora
Data Filtering and Deduplication in Large Language Models: How Heuristics, MinHash LSH, SemDeDup, and Quality Classifiers Curate Pre-Training Corpora Pre-training data quality governs the downstream reasoning, factual accuracy, and sample efficiency of foundation large language models (LLMs). While early pre-training efforts relied on uncurated or lightly processed dumps from web repositories like Common Crawl, modern state-of-the-art models depend on multi-stage data curation pipelines that fi
August 19, 20261 min
Warmup-Stable-Decay (WSD): How Decoupled Annealing Replaced Cosine Decay in Modern LLM Pre-Training
For years, foundation model pre-training adhered to a standard optimization convention: linear learning rate warmup followed by a full-horizon cosine decay. Adopted across GPT-3, PaLM, Chinchilla, and LLaMA, cosine annealing provided stable convergence across diverse parameter scales. However, it introduced a severe structural limitation: the learning rate schedule is rigidly tied to a fixed, upfront token budget. If a team decides to extend pre-training mid-run, branch into domain-specific vari
August 19, 20261 min