Distributed Training12 articles

Distributed Training

Articles

Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning
Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning Training modern large language models requires orchestrating hundreds of billions of parameters across distributed GPU clusters. While parameter counts and optimizer states are fixed for a given model architecture, the activation memory generated during forward passes scales directly with sequence length, batch size, and
August 28, 20261 min
Zero Redundancy Optimizer (ZeRO) and Fully Sharded Data Parallelism (FSDP): Mathematical Foundations of Memory Partitioning, Communication Complexity, and Overlapped Execution
Training modern frontier large language models requires orchestrating compute across thousands of accelerators. As parameter counts scaled from billions to hundreds of billions, the memory capacity of single accelerators (such as 80 GB on an NVIDIA H100 or 96 GB on an H200) became a primary scaling bottleneck. Traditional distributed training paradigms relied on standard Data Parallelism (DDP), where every GPU maintains a complete replica of the model weights, gradients, and optimizer states, sy
August 27, 20261 min
Tensor Parallelism and Pipeline Parallelism: Mathematical Foundations of Megatron-LM 1D Slicing, Sequence Parallelism, and 1F1B Scheduling
Scaling modern large language models beyond tens of billions of parameters quickly exhausts the physical memory and compute throughput of individual graphics processing units. A 70-billion-parameter model stored in 16-bit floating-point (FP16 or BF16) requires 140 GB of VRAM solely for model weights. During full-precision training with the Adam optimizer, parameter states, gradients, and optimizer momentum terms demand approximately 16 to 18 bytes per parameter (amounting to 1.12 TB to 1.26 TB f
August 27, 20261 min
Pipeline Parallelism in Production LLM Training: Comparing 1F1B, Interleaved 1F1B, Zero-Bubble, and DualPipe Schedules
Pipeline Parallelism in Production LLM Training: Comparing 1F1B, Interleaved 1F1B, Zero-Bubble, and DualPipe Schedules Training modern large language models spanning hundreds of billions of parameters requires distributing model layers across multiple compute nodes. While Tensor Parallelism (TP) partitions individual matrix multiplications across GPUs within a single node, its reliance on high-frequency, all-reduce communications limits its practical scaling to the high-bandwidth domain of NVLi
August 25, 20261 min
Ring Attention with Blockwise Transformers: Mathematical Foundations, Circular KV Passing, Overlapped Communication-Computation, and Linear Context Scaling
Scaling the context length of transformer-based large language models has historically collided with two fundamental walls: the $O(S^2)$ computational and memory complexity of standard self-attention, and the high-bandwidth memory (HBM) capacity of individual accelerator devices. While IO-aware tiling algorithms such as FlashAttention eliminate intermediate quadratic activation storage by computing softmax within fast on-chip SRAM, the entire sequence key and value tensors must still reside with
August 25, 20261 min
Cross-Datacenter Distributed LLM Training in Production: DiLoCo, Local SGD, Communication Compression, and High-Latency Fault Tolerance
Cross-Datacenter Distributed LLM Training in Production: DiLoCo, Local SGD, Communication Compression, and High-Latency Fault Tolerance Scaling frontier large language model pre-training within a single datacenter is encountering severe physical limits. Hyperscalers and AI laboratories increasingly face localized power grid saturation, where individual datacenter campuses cannot secure the 500 megawatt to multi-gigawatt utility allocations required for next-generation clusters. Consequently, in
August 24, 20261 min
Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures
Expert Parallelism in Large Language Models: How All-to-All Token Dispatch, Capacity Factors, and Parallel Folding Scale MoE Architectures Scaling dense Large Language Models (LLMs) requires activating every parameter in the network for every token in a sequence. While techniques like Tensor Parallelism, Pipeline Parallelism, and Fully Sharded Data Parallelism distribute billions of dense parameters across clusters, computational cost scales linearly with parameter count. Mixture-of-Experts (Mo
August 19, 20261 min
Activation Checkpointing in Large Language Models: How Selective Recomputation Eliminates Memory Bottlenecks
Large language model pre-training and fine-tuning are fundamentally constrained by GPU memory (VRAM). While distributed techniques such as Fully Sharded Data Parallel (FSDP), ZeRO, and Tensor Parallelism successfully shard model parameters, optimizer states, and gradients across hundreds or thousands of GPUs, activation memory presents a distinct scaling bottleneck. During the forward pass of a transformer model, intermediate tensor outputs must be preserved in GPU memory so that backpropagatio
August 19, 20261 min
Sequence Parallelism in Large Language Models: How Megatron-SP, DeepSpeed Ulysses, and RingAttention Distribute Long Contexts
Sequence Parallelism in Large Language Models: How Megatron-SP, DeepSpeed Ulysses, and RingAttention Distribute Long Contexts Training and serving frontier large language models on context windows spanning hundreds of thousands to millions of tokens introduces a fundamental memory barrier. While model parameters can be distributed across GPUs using Tensor Parallelism (TP) or Fully Sharded Data Parallelism (FSDP / ZeRO), activation memory scales directly with sequence length $S$. For sequence le
August 19, 20261 min
Pipeline Parallelism in Large Language Models: How GPipe, 1F1B Scheduling, and Interleaving Tame Memory and Bubbles
Training frontier large language models with tens or hundreds of billions of parameters exceeds the physical memory capacity of any individual GPU. While intra-node sharding strategies such as Tensor Parallelism partition individual matrix multiplications across accelerators over high-speed NVLink interconnects, scaling across multi-node clusters encounters strict hardware boundaries. Tensor Parallelism requires multiple collective All-Reduce communications per transformer layer. Across standar
August 18, 20261 min
Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training
Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training Training large language models across distributed GPU clusters introduces a fundamental memory bottleneck. In traditional Distributed Data Parallel (DDP) setups, every GPU maintains an identical copy of model weights, optimizer states, and gradients while processing independent data batches. As models scale from billions to hundreds of billions of parameters, static model s
August 18, 20261 min
Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads
Tensor Parallelism in Large Language Models: How Megatron-LM Partitions Multi-Layer Perceptrons and Attention Heads Training and serving modern large language models requires navigating severe hardware memory and compute constraints. While standard Distributed Data Parallelism (DDP) replicates the entire model across multiple accelerators, modern frontier architectures containing tens or hundreds of billions of parameters exceed the physical memory capacity of any single GPU. Even with 80 GB o
August 18, 20261 min