SwiGLU and Gated Linear Units: How Bilinear Gating Replaced Standard FFNs in Modern LLMs

Every modern open-weight and frontier large language model, from Meta's LLaMA 3 and Mistral to Alibaba's Qwen 2.5 and DeepSeek-V3, has abandoned the standard two-layer Feed-Forward Network (FFN) originally introduced in the 2017 Transformer architecture. In its place, model architectures have converged almost universally on Gated Linear Units (GLU), specifically the Swish-Gated Linear Unit (SwiGLU).

While the original Transformer relied on standard non-linear activations like ReLU or Gaussian Error Linear Units (GELU), gated architectures introduce an element-wise multiplicative branching mechanism. This structural modification delivers higher representation capacity and faster training convergence per parameter, fundamentally altering how neural networks process hidden state representations between self-attention layers.

Architectural schematic of the SwiGLU feed-forward network showing dual-pathway gating

The Evolution of Transformer Feed-Forward Networks

In the canonical Transformer architecture described by Vaswani et al. (2017), each layer contains a multi-head attention block followed by a point-wise Feed-Forward Network. The standard FFN consists of two linear transformations separated by a non-linear activation function:

$\text{FFN}(x, W_1, W_2, b_1, b_2) = \sigma(x W_1 + b_1) W_2 + b_2$

Where $x \in \mathbb{R}^{B \times L \times d_{\text{model}}}$ , $W_1 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$ , and $W_2 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$ . In the original formulation, the intermediate hidden dimension $d_{\text{ff}}$ was set to $4 \times d_{\text{model}}$ , and $\sigma$ was the Rectified Linear Unit ( $\text{ReLU}(z) = \max(0, z)$ ).

Subsequent architectures like BERT (Devlin et al., 2018) and GPT-2/GPT-3 (Radford et al., 2019; Brown et al., 2020) replaced ReLU with Gaussian Error Linear Units (Hendrycks & Gimpel, 2016):

$\text{GELU}(z) = z \cdot \Phi(z) = z \cdot P(Z \le z), \quad Z \sim \mathcal{N}(0, 1)$

GELU provided a smooth, non-monotonic approximation that eliminated the hard zero-gradient threshold of ReLU for negative inputs, easing optimization in deep networks. However, the basic structural topology (a single projection up followed by an activation and a projection down) remained unchanged.

Mathematical Formulation of Gated Linear Units

Gated Linear Units were originally proposed by Dauphin et al. (2016) for convolutional language modeling. A classical GLU computes the component-wise product of two linear transformations, one of which is modulated by a sigmoid gating function:

$\text{GLU}(x, W, V, b, c) = \sigma(x W + b) \otimes (x V + c)$

Where $\otimes$ represents the Hadamard (element-wise) product, $W$ acts as the gate projection, and $V$ acts as the value projection.

In 2020, Noam Shazeer published GLU Variants Improve Transformer, exploring variations where the gating function $\sigma$ is replaced with non-linear functions such as ReLU, GELU, and Swish (Ramachandran et al., 2017; Elfwing et al., 2018):

ReGLU: $\text{ReGLU}(x, W, V) = \max(0, x W) \otimes (x V)$
GEGLU: $\text{GEGLU}(x, W, V) = \text{GELU}(x W) \otimes (x V)$
SwiGLU: $\text{SwiGLU}(x, W, V) = \text{Swish}_1(x W) \otimes (x V) = (x W \cdot \text{sigmoid}(x W)) \otimes (x V)$

When integrated into a Transformer FFN layer without bias terms (the modern standard for pre-training stability), the complete SwiGLU feed-forward layer requires three weight matrices:

$\text{FFN}_{\text{SwiGLU}}(x) = \left( \text{SiLU}(x W_{\text{gate}}) \otimes (x W_{\text{up}}) \right) W_{\text{down}}$

Where:

$W_{\text{gate}} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$ projects the hidden state to produce the gating vector.
$W_{\text{up}} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$ projects the hidden state to produce the value signal.
$W_{\text{down}} \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$ projects the modulated representation back to the model hidden dimension.

The 2/3 Dimension Sizing Rule

Replacing a standard two-matrix FFN with a three-matrix GLU introduces an extra projection matrix ( $W_{\text{up}}$ ). If an engineering team retains the classical intermediate expansion factor $d_{\text{ff}} = 4 d_{\text{model}}$ , the total parameter count and floating-point operations (FLOPs) of the FFN increase by 50%.

To maintain compute and parameter parity when benchmarking or migrating architectures, Shazeer proposed scaling the intermediate dimension by a factor of 2/3:

Standard FFN parameters: $2 \times d_{\text{model}} \times d_{\text{ff}} = 2 \times d_{\text{model}} \times (4 d_{\text{model}}) = 8 d_{\text{model}}^2$
SwiGLU FFN parameters: $3 \times d_{\text{model}} \times d_{\text{ff}}$
Parity equation: $3 \times d_{\text{model}} \times d_{\text{ff}} \approx 8 d_{\text{model}}^2 \implies d_{\text{ff}} \approx \frac{8}{3} d_{\text{model}} \approx 2.667 d_{\text{model}}$

In real-world model architectures, this theoretical value is adjusted to conform to hardware tiling constraints. GPU Tensor Cores execute matrix multiplications most efficiently when matrix dimensions are multiples of 64, 128, or 256 bytes.

For instance, in Meta's LLaMA 7B model ( $d_{\text{model}} = 4096$ ):

Theoretical $d_{\text{ff}} = \frac{8}{3} \times 4096 = 10922.67$
LLaMA rounds up to the nearest multiple of 256: $d_{\text{ff}} = 11008$

This deliberate dimension sizing ensures that the computational cost of the SwiGLU FFN matches the original standard FFN while unlocking the performance benefits of multiplicative gating.

Why Gating Outperforms Standard Activations

Shazeer's empirical evaluations demonstrated that across pre-training perplexity and downstream GLUE/SuperGLUE benchmarks, every GLU variant consistently outperformed standard ReLU and GELU layers. Subsequent pre-training at scale, including Google's PaLM (Chowdhery et al., 2022) and Meta's LLaMA (Touvron et al., 2023), confirmed these gains.

Several architectural mechanisms explain why SwiGLU delivers superior representation learning:

1. Multiplicative Gating and Continuous Routing

In a standard FFN, non-linear activation acts as a fixed element-wise threshold. In contrast, the gating projection $x W_{\text{gate}}$ acts as a learned, continuous filter over the linear transformation $x W_{\text{up}}$ . Each feature channel can dynamically suppress, amplify, or invert incoming information based on the token context. This effectively implements soft feature routing at every layer.

2. Gradient Flow and Backpropagation Stability

The gradient of a standard activation layer depends strictly on the derivative of the single activation function $\sigma'(z)$ . For SwiGLU, the gradient with respect to input $x$ decomposes across two coupled pathways:

$\frac{\partial}{\partial x} \left[ \text{SiLU}(x W_{\text{gate}}) \otimes (x W_{\text{up}}) \right] = W_{\text{gate}}^T \left[ \text{SiLU}'(x W_{\text{gate}}) \otimes (x W_{\text{up}}) \right] + W_{\text{up}}^T \left[ \text{SiLU}(x W_{\text{gate}}) \right]$

This dual-pathway gradient flow reduces the likelihood of vanishing gradients in deep networks. Even if the gating signal enters a low-derivative region, the second pathway continues to propagate gradient signal directly through the value projection.

3. Bilinear Expressivity

Mathematically, a gated unit computes a second-order polynomial (bilinear) interaction between projected features. This gives the layer higher expressivity per unit parameter than a standard affine transformation followed by a scalar non-linearity.

Production Systems and Serving Mechanics

While SwiGLU improves convergence and task performance, it introduces systems-level engineering challenges during high-throughput inference and distributed serving:

Kernel Fusion and Memory Bandwidth

A naive PyTorch implementation of SwiGLU executes three separate General Matrix Multiplications (GEMMs), two intermediate memory writes to High Bandwidth Memory (HBM), and an element-wise kernel launch for SiLU and multiplication. Because memory bandwidth, rather than compute, bottlenecks LLM token generation (autoregressive decoding), naive execution incurs severe latency penalties.

High-performance inference engines like vLLM, TensorRT-LLM, and SGLang use fused CUDA and Triton kernels:

Combined Gate-Up GEMM: The gate and up weight matrices are concatenated along the output dimension into a single matrix $W_{\text{gate\_up}} \in \mathbb{R}^{d_{\text{model}} \times 2d_{\text{ff}}}$ .
SRAM-Level Computation: The combined linear projection is computed in a single GEMM call. The resulting intermediate tensors remain in GPU on-chip SRAM registers, where the SiLU activation and Hadamard multiplication execute without round-tripping to global GPU memory (HBM).
Direct Down-Projection: The modulated tensor feeds directly into the down-projection GEMM ( $W_{\text{down}}$ ).

FP8 Quantization and Outlier Dynamics

During FP8 post-training quantization or mixed-precision serving, SwiGLU exhibits distinct numerical properties:

The gate branch ( $\text{SiLU}(x W_{\text{gate}})$ ) is naturally bounded on the negative side (approaching 0 as $z \to -\infty$ ), while the up branch ( $x W_{\text{up}}$ ) is purely linear and unbounded.
Outlier feature activations often concentrate in specific channels of the up-projection. Modern FP8 inference kernels implement per-tensor or per-channel scaling factors independently for the gate and up projections before performing the fused Hadamard product.

Modern Architectural Adoption

SwiGLU and its variants form the standard feed-forward backbone across major modern frontier and open-weight model families:

LLaMA Series (Meta): LLaMA 1, 2, and 3 employ SwiGLU with $d_{\text{ff}} \approx \frac{8}{3} d_{\text{model}}$ padded to multiples of 256.
PaLM and PaLM 2 (Google): Adopted SwiGLU across all parameter scales (up to 540B parameters).
Mistral and Mixtral (Mistral AI): Standardized on SwiGLU for dense models (Mistral 7B) and Mixture-of-Experts routing blocks (Mixtral 8x7B).
Qwen 2 / 2.5 (Alibaba): Implements SwiGLU across its dense and MoE model lines.
DeepSeek-V2 / V3 (DeepSeek): Uses SwiGLU within its fine-grained Mixture-of-Experts routing units.
Gemma / Gemma 2 (Google): Utilizes GEGLU (GELU-Gated Linear Units) with an approximate GELU non-linearity.

By replacing standard single-path activations with gated bilinear projections, modern transformer architectures maximize the parameter efficiency and representational power of feed-forward layers.