Subscribe
AI
5 min read

What Is Gradient Descent? How Neural Networks Find Their Way Downhill

Every large language model you have ever used was trained by the same basic procedure: compute how wrong the model is, then nudge its billions of weights in the direction that makes it less wrong. Repeat until the bill arrives. That nudge has a name, gradient descent, and it is old enough to have been proposed before the internal combustion engine was practical. This explainer covers what gradient descent actually does, why the naive version had to be rebuilt three times to train modern models,

What Is Gradient Descent? How Neural Networks Find Their Way Downhill

Every large language model you have ever used was trained by the same basic procedure: compute how wrong the model is, then nudge its billions of weights in the direction that makes it less wrong. Repeat until the bill arrives. That nudge has a name, gradient descent, and it is old enough to have been proposed before the internal combustion engine was practical.

This explainer covers what gradient descent actually does, why the naive version had to be rebuilt three times to train modern models, and which knobs decide whether a training run works at all. It pairs with our backpropagation explainer: backpropagation computes the gradient, gradient descent uses it.

The core loop

Training a model means minimizing a loss function, a single number that measures how badly the model predicts on a batch of examples. The model's weights are the input to that function; the loss is the output. A well-trained model sits at a low point of the loss surface.

Searching every possible weight setting is impossible. A model with one billion weights has more configurations than atoms in the solar system, and exhaustive, random, and analytical search all fail at that scale. Neural networks instead use gradient-based search, presented by Ian Goodfellow in his CIFAR summer school lecture as the standard option.

The gradient is a vector that points in the direction of steepest increase of the loss. Gradient descent takes the opposite direction: subtract a small multiple of the gradient from every weight, evaluate again, repeat. In code the entire algorithm is one line, weights = weights - learning_rate * gradient. The method works in any number of dimensions, which is why the same loop trains a two-layer classifier and a frontier model.

The idea is not a machine learning invention. Gradient descent is generally attributed to Augustin-Louis Cauchy, who first suggested it in 1847, with Jacques Hadamard proposing a similar method independently in 1907. What made it useful for neural networks was a way to compute the gradient efficiently for layered models, supplied by Rumelhart, Hinton and Williams in their 1986 Nature paper on backpropagating errors. Before 1986 there was a descent method with no cheap way to get the gradient. After 1986 there was a complete training algorithm, and deep learning's modern era followed.

Three flavors, one trade-off

The gradient is an average over training examples, and how many examples you average over before each update defines the classical variants, as Sebastian Ruder's widely cited overview lays out:

  • Batch gradient descent computes the gradient over the entire training set, then takes one step. Stable, but you must pass over all your data before moving an inch. At LLM corpus scale that is not an update rule, it is a lifestyle.
  • Stochastic gradient descent (SGD) updates after a single example. Each step is noisy and often points sideways, but you take billions of steps. Ruder notes it serves as the most basic algorithm used to train deep networks today, and the convergence properties of the nonlinear case were first studied by Haskell Curry in 1944.
  • Mini-batch gradient descent splits the difference: average the gradient over a small random batch, update, repeat. This is what every serious training run uses, because it balances update accuracy against per-step cost and it maps cleanly onto GPU hardware that wants to be fed rectangular matrices.
Mini-batch sampling: a grid of data tiles with a highlighted batch extracted and fed through a cyclic update loop
Step size in gradient descent: a dotted path overshooting across a valley versus one settling into its lowest point

Why the easy version is not enough

Pure mini-batch descent has two practical problems. The first is the learning rate itself, the constant multiplying the gradient. Too small and training crawls. Too large and the steps overshoot, the loss bounces upward, and the run diverges. Choosing it is closer to ritual than theory, and Karpathy's training recipe treats a wrong learning rate as the single most common failure in practice, advising that decays be disabled during debugging precisely because LR schedules quietly kill runs.

The second problem is that one learning rate has to serve every weight. In a deep network, layers see gradients of wildly different magnitudes, so a rate that is right for the embedding matrix may be fifty times too big for the final layer.

The fix history runs through three ideas:

  1. Momentum. Add a fraction of the previous update to the current one, so the optimizer rolls through flat valleys instead of crawling. The heavy-ball formulation goes back to Polyak in 1964 and is covered in Ruder's overview.
  2. Adaptive per-parameter rates. AdaGrad divides each parameter's step by an accumulating estimate of its gradient magnitude, shrinking rates for weights that move often and preserving them for sparse ones. Adam, from Kingma and Ba in 2014, combines that with momentum using exponential moving averages of the first and second gradient moments. The paper's selling points: simple to implement, computationally efficient, invariant to diagonal rescaling of gradients, and suited to problems that are large in data and parameters. That description is a portrait of LLM training.
  3. Decoupled weight decay. The Adam variant that actually trains most modern language models is AdamW, where Loshchilov and Hutter showed that L2 regularization and weight decay are not the same thing once the optimizer is adaptive, and that decoupling the penalty from the gradient update restores its regularization effect.

Descent at LLM scale

Frontier training runs are gradient descent stretched to its limits. The Chinchilla paper, the scaling-law study behind many open models, trains with AdamW, a linear warmup of the learning rate, and a cosine decay schedule, over thousands of GPUs on trillions of tokens. Batch sizes tell the same story of pushing mini-batch trade-offs: LAMB is a layerwise adaptive optimizer built specifically to keep large-batch descent stable, and it scaled BERT pre-training to a batch of 32,768 sequences, cutting training time from three days to 76 minutes on a TPUv3 pod without accuracy loss.

One more myth worth retiring. The old fear of gradient descent was getting stuck in bad local minima. The Deep Learning textbook by Goodfellow, Bengio and Courville explains why that framing is wrong in high dimensions: critical points are dominated by saddle points rather than local minima, and gradient descent is designed to move downhill, not to seek points where the gradient vanishes. In practice the runs that stall do so because of the learning rate, the data, or the batch, not because the optimizer found a comfortable valley.

What to remember

  • Gradient descent minimizes loss by repeatedly stepping weights opposite to the gradient. Cauchy described it in 1847; backpropagation in 1986 made it usable on neural networks.
  • Mini-batch SGD is the workhorse: one gradient per example is too noisy, one per dataset is too slow.
  • The learning rate is the knob that kills runs. Momentum, Adam's per-parameter adaptation, and AdamW's decoupled weight decay are the standard repairs.
  • Modern LLM training is the same loop, just with adaptive optimizers, warmup-and-decay schedules, and batches in the tens of thousands.

There is no secret beyond it. A model is billions of numbers, and training is the patient act of subtracting the slope from each one, a few thousand times a second, for a month.

Sources

AILLMeducational