Subscribe
AI
6 min read

What Is Dropout? Why Training Randomly Switches Off Neurons

For most of the decade from 2012 to 2022, almost every serious neural network shipped with dropout switched on. AlexNet used it. BERT trained with it at a rate of 0.1 (Liu et al., 2019). The public GPT-2 config file still carries attn_pdrop: 0.1 and resid_pdrop: 0.1 (Radford et al., 2019, on Hugging Face). Then frontier models quietly turned it off. The published config for Qwen2.5-7B sets attention_dropout to 0.0 (Yang et al., 2025), and a 2026 study of layer dropout states the trend plainly:

What Is Dropout? Why Training Randomly Switches Off Neurons

For most of the decade from 2012 to 2022, almost every serious neural network shipped with dropout switched on. AlexNet used it. BERT trained with it at a rate of 0.1 (Liu et al., 2019). The public GPT-2 config file still carries attn_pdrop: 0.1 and resid_pdrop: 0.1 (Radford et al., 2019, on Hugging Face).

Then frontier models quietly turned it off. The published config for Qwen2.5-7B sets attention_dropout to 0.0 (Yang et al., 2025), and a 2026 study of layer dropout states the trend plainly: as models and datasets scaled, dropout, particularly layer dropout, "has largely disappeared from large language models pre-training recipes" (Cerebras, 2026).

What was dropout, why did it vanish, and why is some version of it coming back?

The problem dropout attacks: memorizing instead of learning

A neural network with millions of parameters, trained on a limited dataset, can fit the training examples very well and the unseen test examples badly. That is overfitting, and it was the dominant failure mode of deep learning in the early 2010s, when datasets were small and models were already bigger than them.

Geoffrey Hinton and colleagues gave a specific diagnosis in the paper that introduced dropout in 2012. Large networks trained on small data develop "complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors" (Hinton et al., 2012). In practice, some neurons lean on the outputs of particular other neurons, and the whole arrangement only works because those partners happen to be present. The partners are fixed at training time and untrustworthy on new data.

The mechanism: randomly delete units, every step

The fix proposed in the 2012 note and formalized in the 2014 JMLR paper by Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov is blunt. During training, for each example in each mini-batch, temporarily remove ("drop") a random subset of units, along with all their incoming and outgoing connections. Different units are dropped on different steps.

The details, as specified in the original paper:

  • Each hidden unit is kept with probability p (the original paper defines p as a retention probability; modern frameworks usually report the drop rate, 1 minus p). The paper notes that p = 0.5 "seems to be close to optimal for a wide range of networks and tasks," while for input units the optimal retention probability is usually closer to 1.
  • A network with n units can be seen as a collection of 2^n possible "thinned" networks that share weights. Each training step samples a new one.
  • At test time, nothing is dropped. Instead, each unit's outgoing weights are multiplied by p, which makes the single deployed network produce (approximately) the same output as the average of all the thinned networks. Modern implementations fold this scaling into training instead, so inference runs unmodified; that variant is usually called inverted dropout.

The original paper reported the gains concretely. On MNIST, dropout added to L2 regularization cut test error from 1.62 percent to 1.25 percent, and dropout combined with max-norm constraints reached 1.05 percent (Srivastava et al., 2014, Table 9). Dropout also appeared in the AlexNet architecture that won ImageNet classification in 2012 (Krizhevsky, Sutskever, Hinton, NeurIPS 2012).

Small networks drop many units and rely on thinning, while large datasets make dropout largely unnecessary.

Why deleting units makes networks better

The ensemble reading is the clean one. Because the thinned networks share weights and each is trained only rarely, dropout is not literally training an ensemble, but it approximates averaging the predictions of an exponential number of sub-networks, which is what standard ensembling does at far greater cost.

The behavioral reading is what the 2014 paper emphasizes: random dropout "makes the presence of any particular hidden unit unreliable," so a unit cannot rely on specific partners and must learn features that are useful across the combinatorially large variety of contexts in which it might operate. Later work unpacked the mechanism further, showing that dropout acts partly through an implicit effect on optimization, similar in kind to the regularization that comes from small mini-batch noise, and not only through the modified expected loss (Wei, Kakade, and Ma, ICML 2020).

Dropout versus data: why large language models gave it up

Dropout is a remedy for variance, the gap between what the data teaches and what the model memorizes. Its usefulness therefore scales inversely with how much data you have relative to model size.

Google DeepMind's study of data scaling laws in neural machine translation measured exactly this. Training transformers across dataset sizes, the authors "find that the dropout value mostly affects test loss when the dataset size is less than 16M" examples; across regimes, changing dropout moved the scaling exponent only from 0.27 to 0.23, a minor effect (Bansal et al., 2022, ICML).

Pre-training runs for a Chinchilla-optimal LLM feed tens of trillions of tokens to models with far fewer parameters. There is little memorization left for dropout to prevent, and the random masks carry costs: noisier gradients, slower convergence per step, and kernels that do less useful arithmetic per byte moved. Frontier labs dropped it from pre-training. BERT-era settings of 0.1 became a marker of smaller-scale training; GPT-2 shipped with dropout rates of 0.1 while later open-weight models like Qwen2.5 publish configs with dropout at zero.

Where dropout still lives

Dropout did not disappear from applied deep learning; it moved to the settings where the data-to-parameters ratio is small again.

  • Fine-tuning. In the LoRA paper, adaptation runs on modest datasets used nonzero dropout: the DeBERTa XXL experiments list CLS dropout of 0.1 to 0.2 across GLUE tasks and dropout probability 0.1 on E2E and WebNLG generation adaptation (Tables 10 and 11). A small adapter on a few thousand examples is precisely the regime where overfitting is the failure mode.
  • Structured variants. LayerDrop randomly skips entire transformer layers during training, which regularizes and has the side benefit that sub-networks of any depth can be pruned from the one trained model without fine-tuning (Fan et al., ICLR 2020).
  • Consistency objectives. R-Drop runs each training example through two different dropout masks and penalizes the bidirectional KL divergence between the two outputs, forcing the model's many implicit sub-networks to agree. It reported "substantial improvements when applied to fine-tune large-scale pre-trained models" across 18 datasets (Wu et al., NeurIPS 2021).
A stack of transformer layers in which two whole layers are ghosted and the computation bypasses them.

The comeback attempt

The September 2026 Cerebras study "Don't Drop Dropout" argues that unit-level dropout died for defensible reasons but layer-level dropout got abandoned too early (arXiv:2609.05275). Across more than 2,400 training runs from 271M to 8.2B parameters on up to 160B tokens, the authors report that with the right layer distribution, schedule and optimizer settings, layer dropout yields lower loss at equal training FLOPs and saves up to 25 percent of training compute. The trained models also enable inference-time tricks, such as early exit, intermediate-layer skipping and self-speculative decoding, for up to 1.5x speedup with negligible accuracy loss.

That is a claim about layer sparsity, not a revival of dropping 50 percent of hidden units at every step, and no published frontier pre-training recipe has adopted it as of this writing. The economics still rhyme with 2012: dropout came back into the conversation when compute per token was the scarce resource worth trading against variance.

Practical rules of thumb

  • Fine-tuning a model on thousands of examples and losing to the train/validation gap? Dropout (or adapter dropout in LoRA) is a reasonable first knob, around 0.05 to 0.1.
  • Pre-training on trillions of tokens? Leave dropout at zero; your data-to-parameters ratio makes it mostly noise.
  • Training a model you intend to prune or run at variable depth later? Look at LayerDrop and layer-dropout schedules.

Dropout is a good object lesson in how training techniques have shelf lives: a method that solved the dominant failure mode of one era quietly becomes the overhead of the next, and then comes back rebranded when compute gets expensive again.

Sources

AILLMEducation