PyTorch4 articles

PyTorch

Articles

Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning
Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning Training modern large language models requires orchestrating hundreds of billions of parameters across distributed GPU clusters. While parameter counts and optimizer states are fixed for a given model architecture, the activation memory generated during forward passes scales directly with sequence length, batch size, and
August 28, 20261 min
GPU Memory Profiling in Production LLM Serving: CUDA Allocator Internals, PyTorch Snapshots, and VRAM Optimization
In high-throughput large language model serving, memory is the primary constraint governing latency, batch concurrency, and context length. While model parameter footprints are static and easily calculated, runtime GPU memory (VRAM) dynamics are governed by low-level caching allocators, dynamic key-value (KV) cache allocation pools, transient activation spikes, and memory fragmentation. When an inference worker crashes with torch.cuda.OutOfMemoryError, default system diagnostics such as nvidia-
August 22, 20261 min
Distributed Checkpointing in Production: PyTorch DCP, Asynchronous Staging, and Dynamic Topology Resharding
Distributed Checkpointing in Production: PyTorch DCP, Asynchronous Staging, and Topology Resharding In multi-node distributed deep learning, checkpointing is often the largest single source of unforced downtime and degraded Model Flops Utilization (MFU). As models scale to tens or hundreds of billions of parameters across thousands of GPUs, saving model weights and optimizer states using traditional serialization primitives creates severe cluster stalls, memory exhaustion on root ranks, and rig
August 21, 20261 min
Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training
Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training Training large language models across distributed GPU clusters introduces a fundamental memory bottleneck. In traditional Distributed Data Parallel (DDP) setups, every GPU maintains an identical copy of model weights, optimizer states, and gradients while processing independent data batches. As models scale from billions to hundreds of billions of parameters, static model s
August 18, 20261 min

PyTorch

Articles

Activation Checkpointing and Selective Rematerialization: Mathematical Foundations, Sub-Linear Memory Scaling, Recomputation Trade-Offs, and Megatron-LM Attention Partitioning

GPU Memory Profiling in Production LLM Serving: CUDA Allocator Internals, PyTorch Snapshots, and VRAM Optimization

Distributed Checkpointing in Production: PyTorch DCP, Asynchronous Staging, and Dynamic Topology Resharding

Fully Sharded Data Parallel (FSDP) and ZeRO: How Memory Sharding Eliminates Redundant Model States in Distributed Training