
Language model training runs on batches. The model sees a batch of sequences, computes a loss, backpropagates, and takes one optimizer step. The size of that batch is not a stylistic preference. It changes the noise level of the gradient, the largest stable learning rate, and how fast a training job finishes in epochs. It also changes something more mundane: how much GPU memory the run needs.
That last fact is where gradient accumulation comes from. It is the standard trick for getting the update behavior of a large batch while your hardware can only fit a small one.
Why batch size eats memory
During the forward pass, every layer produces activations, intermediate outputs the network has to keep around because backpropagation needs them to compute gradients. The memory cost of those stored activations scales with the number of sequences in the batch. A batch of 32 means 32 sets of activations alive at once. Double the batch and you roughly double that portion of peak memory, on top of the fixed costs of the weights and optimizer state (Daily Dose of DS, ApX ML).
Pretraining recipes want large batches. Your single consumer GPU does not fit them. Historically there were two exits: buy more hardware, or shrink the batch and retune everything around it. Gradient accumulation offers a third.
The trick: delay the update, not the gradient
The mechanism is simple enough to write in five lines of PyTorch:
for i, (x, y) in enumerate(loader):
loss = model(x).loss(y) / accumulation_steps
loss.backward() # gradients add onto param.grad
if (i + 1) % accumulation_steps == 0:
optimizer.step() # one update per K micro-batches
optimizer.zero_grad()You split one big batch into several small "micro-batches." Each micro-batch runs a full forward and backward pass, but instead of applying the gradients immediately, they are summed into a buffer. Only after the last micro-batch does the optimizer take a single step, and the buffer is cleared. The effective batch size is the micro-batch size multiplied by the number of accumulation steps (arXiv 2507.07101).
If you want a batch of 256 but only 64 sequences fit in memory, you run four micro-batches of 64 and get a gradient equivalent to one batch of 256 (Sebastian Raschka).
One detail makes the equivalence work: the weights must not update between micro-batches. Every micro-batch has to compute its gradient at the same parameter values, exactly as it would have if all 256 sequences had gone through the model together. Skipping the optimizer step, rather than merely deferring it and stepping in between, is what keeps the sum honest (Medium / Data Science).
Where did this become standard practice? The lineage runs through BERT. Google's 2018 reference fine-tuning implementation shipped a gradient_accumulation_steps parameter, and researchers with single 16 GB GPUs set it to 4 or 8 to match the paper's intended effective batch sizes within their memory limits (M. Brenndoerfer). Today the knob is a first-class option in every major training framework, including Hugging Face Accelerate (HF docs).
What it buys you, and what it does not
Gradient accumulation buys memory headroom, not speed. You still perform the same number of forward and backward passes. Your GPU processes 64 sequences four times instead of 256 once, so wall-clock time per optimizer step roughly quadruples. The per-sample compute is identical.
The technique also quietly changes your schedule. If one optimizer step consumes four micro-batches, a learning-rate schedule counted in optimizer steps now covers four times as much data per step. Schedules specified in "steps" advance more slowly in terms of tokens seen, which matters when you port a recipe between accumulation settings (M. Brenndoerfer).
There is also a live argument about whether the large effective batch is worth it at all. A July 2025 study, "Small Batch Size Training for Language Models," found that language models train stably down to batch size one when Adam's hyperparameters are scaled properly, that small batches were more robust to hyperparameter choices, and that they achieved equal or better performance per FLOP (arXiv 2507.07101). The authors recommend against gradient accumulation unless you are training on multiple devices with model replicas, and suggest a simpler default: use the smallest batch size that maximizes throughput. The conventional wisdom it challenges is real, though. Large batches average away gradient noise, which is why frontier recipes have favored them for years.
Two ways the math goes subtly wrong
When practitioners say gradient accumulation is "equivalent" to a big batch, two caveats deserve attention.
The padding bug. Language model batches contain sequences of different lengths, padded to fit a rectangle. Standard accumulation averages per-batch losses, but if one micro-batch is half padding tokens and another is nearly full, averaging their losses weights them unevenly. The correct causal-LM loss under accumulation is the total loss across all micro-batches divided by the total number of non-padded tokens in them, which is not the same as the mean of the per-batch means. A 2023 investigation found this error in widely used trainer implementations (Kaitchup), and Hugging Face now documents the correction in its Accelerate guide (HF docs). Pretraining mostly escapes the issue because packed sequences rarely carry padding.
The rounding bias. Accumulation means the optimizer receives many tiny updates at low precision. The 2025 small-batch study notes that when master weights are stored in bfloat16, standard deterministic rounding can bias weight updates, so accumulating gradients across many small batches drifts from what one larger batch would compute. Computing the update in float32, or applying stochastic rounding, removes the bias (arXiv 2507.07101v4).
The mental model
Think of gradient accumulation as a poll with a delayed count. Each micro-batch is a precinct that casts its votes, but nothing gets reported until the last precinct finishes. The tally equals what a single big precinct would have produced, as long as nobody changes the ballot rules (the weights) while counting. It is not faster than counting everyone at once. It just fits in a smaller room.
For a practitioner the decision reduces to three numbers: micro-batch size (what fits in memory), accumulation steps (how much bigger you want the batch), and effective batch size (their product). If quality says the effective batch should be 256 and memory says 64, accumulation steps are 4. Everything else in the recipe, learning rate, warmup, schedule length, is then specified in terms of the effective batch, not the micro-batch.
Sources
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful, Marek et al., July 2025 (v4 December 2025)
- Performing gradient accumulation with Accelerate, Hugging Face documentation
- Single-GPU LLM Finetuning, Sebastian Raschka, 2023
- Fixing Faulty Gradient Accumulation, Kaicheng Zhang
- Gradient Accumulation in Practice, M. Brenndoerfer
- What is Gradient Accumulation in Deep Learning?, Medium
- Memory Optimization During LLM Fine-tuning, ApX ML
- Gradient Accumulation in Neural Networks and How it Works, Daily Dose of Data Science
