Training8 articles

Training

Articles

Sequence Packing and Loss Masking in Production SFT: Architecture, Trade-Offs, and Framework Implementations
Sequence Packing and Loss Masking in Production SFT: Architecture, Trade-Offs, and Framework Implementations Supervised fine-tuning pipelines routinely waste 40–70% of FLOPs on padding tokens. Sequence packing eliminates that waste by concatenating multiple training examples into a single long tensor, but it introduces three hard constraints: attention must not leak across example boundaries, loss must be computed only on the intended target spans (typically assistant responses), and the data c
August 24, 20261 min
Gradient Low-Rank Projection (GaLore): How Dynamic Subspaces Enable Full-Parameter LLM Pre-Training with Low-Rank Memory
Pre-training foundation large language models has historically required massive compute clusters, largely due to the memory footprint of optimizer states rather than the model weights themselves. While parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) freeze weights and introduce small adapter matrices, they fail when applied to pre-training from scratch because they restrict parameter updates to a fixed, static low-rank manifold. Gradient Low-Rank Projection (GaLore),
August 23, 20261 min
Chunked and Fused Cross-Entropy: How Online Logit Tiling Slashes Large-Vocabulary VRAM Bottlenecks in LLM Training
Chunked and Fused Cross-Entropy: How Online Logit Tiling Slashes Large-Vocabulary VRAM Bottlenecks in LLM Training As frontier large language models have scaled, tokenizer vocabularies have expanded substantially. Where early architectures such as LLaMA and Mistral relied on 32,000 subword tokens, contemporary models routinely employ vocabularies of 128,256 tokens (Llama 3), 152,064 tokens (Qwen 2.5), and 256,000 tokens (Gemma 2). Larger vocabularies compress text more densely, improve multilin
August 21, 20261 min
AI Data Startup Micro1 Reaches 00M Gross Run Rate Amid Training Demand
Four-year-old AI data and annotation startup Micro1 has reached a $500 million gross annualized run rate, expanding fivefold from $100 million eight months ago as foundation model builders scale spending on post-training datasets and reinforcement learning environments. After accounting for contractor compensation paid to specialized annotators, Micro1 retains approximately 60% to 70% of gross billings, placing its net annual run rate between $150 million and $200 million. The Shift Toward Ex
August 21, 20261 min
Maximal Update Parametrization (muP): How Tensor Programs Enable Zero-Shot Hyperparameter Transfer in LLM Pre-Training
Pre-training a frontier large language model requires hundreds of thousands of GPU hours and millions of dollars in compute. At that scale, traditional hyperparameter tuning is financially and operationally impossible: teams cannot sweep learning rates, weight initializations, or optimizer betas across multiple 70B parameter runs to find the loss minimum. Historically, practitioners relied on ad-hoc heuristic extrapolation or manual guesses from small runs, often leading to sub-optimal loss curv
August 20, 20261 min
Muon Optimizer: How Matrix Orthogonalization and Newton-Schulz Iterations Accelerate LLM Training
Modern large language model pre-training has relied on AdamW as its default optimizer for nearly a decade. While AdamW provides robust convergence across varied architectures, its fundamental formulation treats neural network weights as flat collections of independent scalar parameters. For the 2D weight matrices that dominate Transformer architectures—including attention projections and feed-forward linear layers—this coordinate-wise treatment ignores the underlying matrix geometry and singular
August 19, 20261 min
FP8 Mixed-Precision Training: Formats, Scaling Recipes, and Fine-Grained GEMMs
FP8 Mixed-Precision Training: Formats, Scaling Recipes, and Fine-Grained GEMMs Training frontier large language models requires navigating harsh memory bandwidth and compute boundaries. For years, 16-bit formats such as IEEE FP16 and Brain Floating Point (BF16) served as the standard arithmetic precision for deep learning workloads. While BF16 provided sufficient dynamic range to prevent gradient underflow without manual loss scaling, training clusters still spent substantial time bounded by me
August 19, 20261 min
Multi-Token Prediction: How Future Token Supervision Densifies Representations and Speeds Up LLM Serving
Standard autoregressive language models are trained under a strict next-token prediction objective. At every sequence position, the model consumes a prefix of tokens and predicts the single immediate successor token using a cross-entropy loss. While this paradigm has scaled language modeling across orders of magnitude, it suffers from an architectural limitation: myopic optimization. By evaluating loss exclusively on the immediate next step, standard training fails to reward representations that
August 18, 20261 min