Subscribe
educational
6 min read

What Is an Activation Function? The Kink That Keeps Neural Networks From Collapsing

Sigmoid, ReLU, GELU, SwiGLU: a one-line function decides whether a stack of matrices can learn anything. Here is how the choice changed deep learning, twice.

Retro print-style cover: What Is an Activation Function, with plotted curves and gradient arrows
Retro print-style cover: What Is an Activation Function, with plotted curves and gradient arrows

What Is an Activation Function? The Kink That Keeps Neural Networks From Collapsing

A neural network is mostly a stack of matrix multiplications. Between every pair of matrices sits one small, strange operation: the activation function. It looks like a formality, a single nonlinearity squeezed between the "real" work of weights. It is not a formality. Without it, the entire stack collapses into one matrix, and the network is no smarter than a spreadsheet formula.

The collapse problem

Each layer of a network computes a linear map: multiply the input vector by a weight matrix, add a bias. Linear maps compose into linear maps. Pass a vector through ten layers of pure multiplication and you could have replaced all ten weight matrices with a single precomputed matrix. Deeper would buy nothing.

The activation function is what breaks this. Applied element-wise after each layer, it makes the map nonlinear, so composition stops collapsing. The choice of which nonlinearity determines three things that matter more than anyone expected in the 1990s: how fast training converges, how deep a network can get before gradients die, and how cheaply the finished model runs.

Frank Rosenblatt's 1958 perceptron used a step function: fire if above threshold, stay silent if below (Rosenblatt, 1958). A step has zero derivative almost everywhere, so there is nothing for gradient descent to follow. When Rumelhart, Hinton and Williams popularized backpropagation in 1986 (Nature, 1986), they needed smooth, differentiable functions, and the field settled on the logistic sigmoid and tanh.

Sigmoid: the function that strangled depth

The sigmoid squashes any input into the range 0 to 1, which suited binary neurons and probability outputs. It also has a defect that took a decade to diagnose. Its derivative is σ(x)(1−σ(x)), which never exceeds 0.25. Push the input far from zero and the curve flattens; the neuron saturates and its gradient vanishes.

Backpropagation works by multiplying derivatives across layers, chain-rule style. With sigmoid everywhere, a signal heading backward through a deep stack is multiplied by numbers no larger than 0.25 at every layer. By layer ten the gradient is astronomically small. Early weights stop learning entirely. This is the vanishing gradient problem, and it is the reason the 1990s produced shallow two-layer nets and a decade of quiet (Berkeley CS189 lecture notes walk through the arithmetic).

A saturating S-curve next to a flat-then-linear kinked curve

ReLU: the kink that unlocked deep learning

In 2010, Vinod Nair and Geoffrey Hinton replaced the sigmoid in restricted Boltzmann machines with the simplest nonlinearity they could find: max(0, x) (Nair & Hinton, ICML 2010). Negative inputs go to zero. Positive inputs pass through unchanged. The function became the Rectified Linear Unit, ReLU.

The properties that mattered:

  • No saturation on the positive side. The derivative is exactly 1 for positive inputs, so backpropagated gradients stop shrinking there. Depth becomes trainable.
  • Cheap. One comparison per element, no exponentials.
  • Sparse. Half the units in a layer output zero for any given input, which the authors argued made networks learn parts of the data that fit each input.

AlexNet, the network that won ImageNet in 2012, put the practical claim in writing: deep convolutional networks with ReLUs train several times faster with the same momentum schedule than equivalents using saturating units (Krizhevsky, Sutskever & Hinton, NIPS 2012). ReLU became the default for the next five years of deep learning.

The kink has a cost. A unit whose input goes negative emits zero gradient, so if bad luck or a high learning rate pushes it permanently negative, it never recovers. Practitioners call this the dying ReLU problem. Andrew Maas, Awni Hannun and Andrew Ng's fix was to let the negative side leak: give it a small slope instead of zero (Maas et al., ICML 2013 workshop). Their leaky ReLU improved word error rates in speech models, by roughly 2% absolute over sigmoid baselines in the pretraining-free setting they tested.

The smoothing era: ELU, GELU, Swish

ReLU's hard kink at zero complicates optimization and kills negative information entirely. The 2015-2017 wave of successors was a quest for something smooth where ReLU is sharp, and non-saturating where sigmoid is fatal.

  • ELU (2015) mirrors the positive side with an exponential decay toward −1 for negative inputs (Clevert et al., arXiv:1511.07289).
  • GELU (2016) gates the input by its own value under a Gaussian: xΦ(x). Where ReLU asks "is x positive?", GELU asks "how strongly positive?", and passes small positive values through partially (Hendrycks & Gimpel, arXiv:1606.08415). A tanh approximation made it fast enough for production.
  • Swish/SwishRL (2017) came out of an actual search: Ramachandran, Zoph and Le ran an overparameterized meta-learning experiment over candidate activations and landed on x·σ(x), which outperformed ReLU at scale (arXiv:1712.05855). The dip just below zero on negative inputs turned out to help.

Transformers arrived in 2017 using plain ReLU in their feed-forward blocks (Vaswani et al., arXiv:1706.03762). By the time the scaling era began, GPT-2's release configuration switched that ReLU to a tanh-approximated GELU (GPT-2 model card, gelu_new), and GELU became the standard Transformer feed-forward activation for several years.

SwiGLU: gating beats squashing

The latest regime change started with a question that is not about the curve shape at all. Noam Shazeer's 2020 note asked: why apply the activation to a single projection when you can apply it to one projection and use it as a multiplicative gate on a second? Gated Linear Units multiply two streams element-wise, one of them passed through a nonlinearity. Swapping the sigmoid for Swish or GELU in that gating stream gave ReGLU, SwiGLU and GEGLU (Shazeer, arXiv:2002.05202).

Two signal streams, one gated, meeting at a multiply node

In parameter- and compute-matched T5 experiments, GEGLU and SwiGLU beat the ReLU baseline on pre-training perplexity and on SuperGLUE (aggregate 74.56 for SwiGLU versus 72.76 for the ReLU FFN). The cost: the FFN now has three weight matrices instead of two, so Shazeer's recipe shrinks the hidden dimension to keep the parameter count equal.

Meta's LLaMA paper adopted "the SwiGLU activation function, introduced by Shazeer (2020), to improve the performance" as one of its four architectural deviations from the original Transformer (Touvron et al., arXiv:2302.13971), and Google's PaLM did the same (Chowdhery et al., arXiv:2204.02311). Mistral and most open-weight successors followed. If you use a 2026-era chat model, its largest block of parameters multiplies two streams through a smooth gate.

The kink never actually died

One loose end from the ReLU era turned out to be a feature. Because ReLU outputs exactly zero for negative inputs, a large fraction of a network's activations are zero for any given token, and dead-simple hardware can skip the multiply. A 2023 study exploited this activation sparsity: keeping plain ReLU in large models and using the resulting ~sparsity for inference speedups, while showing ReLU remains competitive with gated variants at large batch sizes (Abatan et al., arXiv:2310.04564). The same paper notes Falcon and MPT stayed on GELU while LLaMA and PaLM went gating, an accurate snapshot of the split in 2023, and it persists today.

The practical takeaway for anyone reading model configs: the hidden_act or FFN line in a model card is not boilerplate. It records a decision about trainability (does the gradient survive depth), quality (how smoothly can the network fit functions), and serving cost (how much of each multiply you can skip). Sigmoid bought differentiability and cost you depth. ReLU bought depth and cost you dead units. GELU bought smoothness and cost you an exponential approximation. SwiGLU bought accuracy and costs you a third matrix in every feed-forward block. Every one of them is, at bottom, one line of code between two matrices.

Sources

educational