Every frontier training run starts from a config file full of numbers. One of them matters more than the rest: the learning rate. It is a single scalar, usually written in scientific notation somewhere between 0.00003 and 0.001, that controls how aggressively the model rewrites its own weights every step. Get it roughly right and you spend months of GPU time training. Get it wrong and you spend that same money watching a loss curve flatline or explode.
This is a companion to our gradient descent explainer, which covered the direction the model moves in. This one covers the size of the step.
The step size in an optimizer
A neural network learns by repeated small corrections. Each training step, the model computes a gradient: a vector that points uphill in the loss function, telling you which way to nudge every parameter to make the loss worse. The optimizer flips that vector and moves the parameters in the opposite direction. The learning rate is the multiplier on that move. Double it and every parameter takes twice the step in the same direction; halve it and the model inches along.
In its plain form, that is all there is to it. One caveat for modern practice: large language models are trained with Adam or AdamW, adaptive optimizers that rescale the gradient per parameter before the learning rate is applied. Roughly speaking, Adam normalizes each step to a size proportional to the learning rate regardless of how big or small the raw gradient is. In practice, an Adam step moves each weight by about the learning rate itself, typically a few ten-thousandths. That is why peak learning rates for LLMs cluster in such a narrow band, and why the value that works for one model size can be badly wrong for another.
Too big, too small, and just right
A learning rate that is too high makes updates that overshoot the minimum. The model bounces past good parameter settings, the loss curve spikes, and past a certain point training diverges outright and the run is dead. A learning rate that is too small never diverges and also never finishes: each step moves the weights so little that the model burns its entire compute budget crawling.
Andrej Karpathy's widely cited training recipe puts it bluntly: the learning rate is the hyperparameter that "affects all aspects of training the most" and it should be "beaten with a stick" until it gives its best. His debugging advice to watch for spikes and suspiciously flat curves is standard practice in every lab that trains models at scale.
There is a sharper concept behind this: every training setup has a maximum usable learning rate. Push past it and training destabilizes. The critical learning rate depends on batch size: bigger batches give less noisy gradient estimates, which can support bigger steps. That relationship was formalized by Facebook AI Research in 2017 in the paper that trained ImageNet in one hour. Its linear scaling rule says that when you multiply the minibatch size by k, you multiply the learning rate by k, and it introduced the gradual warmup schedule that every LLM training run still uses (more on that below).

What actual models use
Labs publish these numbers, and they show a consistent pattern: bigger models train at smaller learning rates.
GPT-3, the 175B model OpenAI released in 2020, used a peak learning rate of 0.00006, one tenth of what its smallest 125M variant used, with cosine decay down to 10 percent of peak over the run. Meta's Llama 3 family in 2024 followed the same slope: 0.0003 peak for the 8B model, 0.00015 for 70B, and 0.00008 for 405B, which trained on AdamW with 8,000 warmup steps and a cosine decay to 0.0000008 over 1.2 million steps.
Why does scale shrink the step? Intuitively, a model with hundreds of billions of parameters stores much more per weight, so the same absolute update is more disruptive. The deeper account is incomplete; this is one of those places where neural network training is still an experimental science, and labs pick values from scaling-law sweeps rather than from theory.
Fine-tuning works at the other end of the range. Adapting an already trained model typically uses learning rates one or two orders of magnitude below pre-training peaks, small enough to specialize the model without shredding what pre-training built.
The schedule matters as much as the number
No serious run trains at one constant learning rate. The value changes on a curve over the run, and the shape of that curve is its own subfield.
Warmup. Start near zero and ramp linearly to the peak over the first stretch of steps. Goyal et al. introduced gradual warmup for large-batch SGD in 2017 because the scaled-up learning rate destabilizes early training; the original 2017 Transformer paper used it for the same reason. In LLM runs warmup covers a small slice of the schedule: 8,000 steps of 1.2 million for Llama 3 405B, about 0.7 percent. A common practical justification is that Adam's internal per-parameter statistics need thousands of steps to become meaningful before full-size steps are safe.
Cosine decay. After the peak, the learning rate follows a half-cosine down toward a floor, usually around one tenth of peak, following the convention established by the Chinchilla scaling-laws report in 2021. This was the default from GPT-3 through Llama 2, Llama 3 and Mistral 7B. Its weakness is economic: cosine needs the total step budget fixed upfront, because the curve's length is part of the definition. Cut a run short or extend it and the schedule is wrong.
Warmup-stable-decay (WSD). The MiniCPM group's 2024 answer to that weakness. Ramp to peak, hold it flat indefinitely, then drop it once you decide to stop, branching a decaying copy off the main constant-rate line. DeepSeek-V3 and Kimi K2 trained this way. A 2024 theoretical analysis by Wen, Liang, Ma and colleagues tied WSD's behavior to a "river valley" loss landscape: at high constant rates iterates drift along a low-loss valley floor instead of converging to a point, and decay is the move down into the valley.
The 2026 twist. Sebastian Raschka's November 2026 survey of published training recipes found a new camp: no decay at all. Kolibri-1 and DatologyAI's models train at a constant learning rate and average the last checkpoints instead of annealing. Cosine remains the majority choice (Llama 3, OLMo 2, GLM, Phi-4), WSD is the second standard (DeepSeek V3, Kimi K2, SmolLM3, MiniMax M2), and constant-plus-averaging is the newcomer. Meanwhile a January 2025 study, "Straight to Zero," reported that plain linear decay to zero matched or beat cosine, which is a reminder that even the "settled" parts of this recipe are contested.

Why nobody can compute it for you
The uncomfortable truth about the learning rate is that it has no reliable first-principles theory. You cannot derive the right value from the architecture and the data. Labs find it the way the field has always found it: run small proxy models across a grid of values, fit which peak works at each scale, and hope the pattern extrapolates to the full run that costs tens of millions of dollars. The scheduler, by contrast, is now a genuine design decision: whether you commit to a step budget upfront (cosine) or keep your options open until the money runs out (WSD).
If you take one thing from this piece, take this: the learning rate is not a knob you set once. It is a plan. A run is a trajectory of step sizes, warmup in, decay out, and the shape of that trajectory is as much a part of a frontier model's recipe as the architecture itself.
Sources
- Brown et al., "Language Models are Few-Shot Learners" (GPT-3), 2020 - https://arxiv.org/abs/2005.14165
- The Llama 3 Herd of Models, Meta AI, 2024 - https://arxiv.org/abs/2407.21783
- Goyal et al., "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour", 2017 - https://arxiv.org/abs/1706.02677
- Vaswani et al., "Attention Is All You Need", 2017 - https://arxiv.org/abs/1706.03762
- Karpathy, "A Recipe for Training Neural Networks", 2019 - https://karpathy.github.io/2019/04/25/recipe/
- Rae et al., "Scaling Language Models: Methods, Analysis & Insights from Training Gopher" (Chinchilla), 2021 - https://arxiv.org/abs/2203.15556
- Wen, Li, Wang, Hall, Liang, Ma, "Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective", 2024 - https://arxiv.org/abs/2410.05192
- Hu et al., "MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies" (source of WSD), 2024 - https://arxiv.org/abs/2404.06395
- DeepSeek-V3 Technical Report, 2024 - https://arxiv.org/abs/2412.19437
- Sebastian Raschka, "Pre-Training Without Learning Rate Decay", November 2026 - https://sebastianraschka.com/blog/2026/llm-pretraining-without-learning-rate-decay.html
- Sawade & Schnekenburger, "Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs", 2025 - https://arxiv.org/abs/2501.18965
- zeroentropy.dev, "Learning-rate scheduler: warmup, cosine decay, and WSD for LLMs" - https://zeroentropy.dev/concepts/learning-rate-scheduler
