Emergent Outlier Features in Large Language Models: Why Hidden Dimension Spikes Arise at Scale and How They Reshape Quantization

When language models scale past approximately 6.7 billion parameters, their internal representations undergo a sharp qualitative phase transition. In smaller models (125M to 2.7B parameters), hidden state activations remain relatively compact, bounded within predictable normal distributions across all embedding dimensions. However, as demonstrated by Dettmers et al. (2022), scaling beyond the 6.7B threshold causes high-magnitude activation spikes—reaching absolute values between 50 and 100, compared to standard baseline activations between -2 and 2—to emerge consistently across specific coordinate channels.

These emergent outlier features appear in less than 0.1% of hidden dimensions, yet they govern the model's predictive capability and long-range coordination. When engineers first attempted to compress 7B+ models using standard 8-bit integer (INT8) quantization, models suffered catastrophic degradation: perplexity exploded, and output degraded into gibberish. Understanding why these outlier channels emerge, how they function mechanistically within transformer layers, and how modern inference runtimes preserve or eliminate them is essential to modern LLM engineering.

The 6.7B Parameter Phase Shift

The emergence of outlier features was first characterized in BERT models by Kovaleva et al. (2021), who identified specific coordinate dimensions with extreme activation norms that disproportionately influenced attention distributions. However, their systematic investigation across autoregressive decoder models at scale was formalized by Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer in their seminal work on LLM.int8() (2022).

Dettmers et al. tracked activation distributions across the OPT model family (from 125M to 175B parameters) and the BLOOM series. Their empirical measurements revealed three distinct operational regimes:

1. Sub-Threshold Scaling (< 2.7B parameters): Outlier activations occasionally appear in isolated token positions or shallow attention layers, but they remain transient and do not persist across the entire transformer stack. Standard uniform quantization methods succeed without special handling.
2. Phase Transition Region (2.7B to 6.7B parameters): Outlier features begin to stabilize. Their frequency of occurrence increases as an exponential function of model perplexity rather than raw parameter count alone.
3. Emergent Regime (6.7B+ parameters): Outlier dimensions become fully systematic. In 100% of transformer layers, a tiny subset of coordinate dimensions (typically 6 to 15 channels out of 4,096 or 8,192 hidden dimensions) exhibit extreme magnitudes exceeding 6.0 standard deviations across virtually all sequence tokens.

The emergence is strictly coupled to the model reaching a specific perplexity threshold. When a smaller transformer is over-trained on massive token corpora until it reaches comparable perplexity to larger models, identical outlier dynamics begin to manifest.

Mathematical Anatomy of Outlier Channels

In a standard transformer residual stream, hidden state vectors $x \in \mathbb{R}^{d_{model}}$ are passed sequentially through multi-head self-attention (MHA) and feed-forward network (FFN) blocks:

$$x_{l+1} = x_l + \text{MHA}(\text{LN}(x_l)) + \text{FFN}(\text{LN}(x_l + \text{MHA}(\text{LN}(x_l))))$$

In sub-6B models, the components of $x_l$ behave as zero-mean, unit-variance Gaussian variables across sequence length $T$ and hidden dimension $d$:

$$x_{i,j} \sim \mathcal{N}(0, \sigma^2), \quad |x_{i,j}| \le 3.5\sigma \approx [-2.5, +2.5]$$

In models exceeding 6.7B parameters, the activation tensor $X \in \mathbb{R}^{T \times d_{model}}$ develops severe coordinate-aligned anisotropy. For a tiny subset of indices $J_{outlier} \subset \{1, \dots, d_{model}\}$ where $|J_{outlier}| \ll d_{model}$:

$$\forall j \in J_{outlier}: \quad |x_{t,j}| \in [20.0, 100.0] \quad \text{for } t \in \{1, \dots, T\}$$

$$\forall j \notin J_{outlier}: \quad |x_{t,j}| \in [-2.0, +2.0]$$

Crucially, these outliers are coordinate-aligned: the extreme values do not jump randomly across different hidden dimensions from token to token. Instead, dimension $j = 142$ or dimension $j = 891$ will consistently carry the 50x to 100x activation spike across all sequence positions and throughout every layer in the network.

Mechanistic Causes: Why Do Outliers Emerge?

Mechanistic interpretability research, including analyses by Bondarenko et al. (2021) and Xiao et al. (2023), explains why transformer optimization drives the formation of outlier channels:

1. Softmax Attention Saturation and Attention Sinks

In self-attention, query and key dot-products are normalized via softmax:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

When an attention head does not find relevant context for a given token, it requires a "no-op" or neutral state. Because softmax rows must strictly sum to 1.0, the model cannot output an attention probability of zero across all tokens. Instead, attention heads learn to dump excess attention probability into initial sequence tokens (the attention sink phenomenon) or dedicated coordinate dimensions in the key/value projections. Large-magnitude feature spikes allow individual attention heads to saturate the softmax function, forcing attention weights to 1.0 on designated reference tokens and 0.0 elsewhere.

2. Syntactic and Delimiter Tracking

Ablation studies reveal that outlier dimensions correlate strongly with structural punctuation tokens (periods, commas, line breaks) and rare lexical items. The model dedicates specific coordinate axes to tracking global sentence boundaries and syntactic state across deep sequence contexts. A single high-magnitude coordinate channel acts as a global broadcast channel within the additive residual stream, allowing subsequent layers to read syntactic status without interference from lower-magnitude semantic embeddings.

3. Non-Linear Gating in Feed-Forward Sublayers

In modern SwiGLU or GELU feed-forward networks:

$$\text{FFN}(x) = (xW_{gate} \odot \text{swish}(xW_{up}))W_{down}$$

The gating mechanism suppresses noise by zeroing out negative activations. Extreme positive activations along dedicated dimensions allow the FFN to selectively ungate complex subnetworks with deterministic precision, ensuring that critical routing decisions dominate across deep transformer layers.

Why Outliers Devastate Uniform Quantization

Standard uniform post-training quantization (PTQ) maps floating-point activations $X \in \mathbb{R}$ to $b$-bit signed integers $X_{int} \in [-2^{b-1}, 2^{b-1}-1]$ (for INT8, $[-128, 127]$):

$$X_{int} = \text{clamp}\left(\left\lfloor \frac{X}{\Delta} \right\rceil, -128, 127\right), \quad \Delta = \frac{\max(|X|)}{127}$$

Where $\Delta$ is the quantization scale factor (step size).

When an activation tensor contains standard values in the range $[-2, +2]$ alongside an outlier spike of $+100.0$:

$$\Delta = \frac{100.0}{127} \approx 0.7874$$

Under this scaling factor:

• An activation of $1.5$ quantizes to $\lfloor 1.5 / 0.7874 \rceil = \lfloor 1.905 \rceil = 2$, dequantizing back to $2 \times 0.7874 = 1.575$ (a 5.0% relative error).
• An activation of $0.3$ quantizes to $\lfloor 0.3 / 0.7874 \rceil = \lfloor 0.381 \rceil = 0$, completely erasing the signal (a 100% relative error).

Because $\Delta$ is stretched across the entire range to accommodate $+100.0$, the discrete quantization bins become wider than the subtle differences between normal features. The 99.9% of features that carry semantic context are crushed into zero or adjacent integer bins, destroying model accuracy.

Algorithmic Solutions Across Inference Engines

To deploy large models at low precision without degradation, researchers developed three primary architectural paradigms:

1. Vector-Wise Mixed-Precision Decomposition (LLM.int8())

Dettmers et al. (2022) introduced mixed-precision matrix multiplication. For an input activation matrix $X \in \mathbb{R}^{T \times d_{in}}$ and weight matrix $W \in \mathbb{R}^{d_{in} \times d_{out}}$:

1. Identify outlier column indices where $\max_t |X_{t,j}| \ge \alpha$ (threshold $\alpha \approx 6.0$).
2. Split $X$ into an outlier submatrix $X_{outlier} \in \mathbb{R}^{T \times k}$ (with $k \ll d_{in}$) and regular submatrix $X_{dense} \in \mathbb{R}^{T \times (d_{in}-k)}$.
3. Split $W$ into corresponding rows $W_{outlier} \in \mathbb{R}^{k \times d_{out}}$ and $W_{dense} \in \mathbb{R}^{(d_{in}-k) \times d_{out}}$.
4. Compute:

$$Y = \underbrace{X_{outlier} W_{outlier}}_{\text{FP16 GEMM}} + \underbrace{\text{dequant}(\text{quant}(X_{dense}) \cdot \text{quant}(W_{dense}))}_{\text{INT8 GEMM}}$$

Because $k$ is tiny (e.g., 6 columns out of 4,096), the FP16 matrix multiplication accounts for less than 0.2% of total FLOPs while preserving zero quantization error for the outlier channels.

Input Activation X (T x d)              Weight Matrix W (d x d_out)
┌─────────────────────────────────┐      ┌───────────────────────────────┐
│       | O |                     │      │                               │
│       | u |                     │      ├───────────────────────────────┤  <- Outlier Row
│       | t |     Dense INT8      │  x   │       Outlier FP16 Row        │
│       | l |                     │      ├───────────────────────────────┤
│       | i |                     │      │                               │
│       | e |                     │      │          Dense INT8           │
│       | r |                     │      │                               │
└─────────────────────────────────┘      └───────────────────────────────┘
         ^ Outlier Col (FP16)

2. Cross-Layer Mathematical Migration (SmoothQuant)

Rather than splitting execution paths into mixed-precision kernels, Xiao et al. (2023) observed that linear layers in transformers can absorb invertible diagonal scale transformations:

$$Y = X W = (X \cdot \text{diag}(s)^{-1}) \cdot (\text{diag}(s) \cdot W) = \hat{X} \hat{W}$$

SmoothQuant calculates per-channel smoothing factors $s_j \in \mathbb{R}^{d_{in}}$:

$$s_j = \frac{\max_t |X_{:, j}|^\alpha}{\max_k |W_{j, :}|^{1-\alpha}}$$

By choosing migration hyperparameter $\alpha = 0.5$, difficulty is split equally between activations and weights. Outlier spikes in activations are scaled down by dividing by $s_j$, while the corresponding weight rows are multiplied by $s_j$. Because weights have static distributions with low dynamic range, they easily absorb the scaled magnitude, enabling uniform INT8 per-tensor activation and weight quantization (W8A8).

3. Coordinate Incoherence and Randomized Rotations (QuaRot & SpinQuant)

The most recent advance in outlier elimination bypasses both mixed precision and per-channel migration. As proven by Ashkboos et al. (2024) in QuaRot and Liu et al. (2024) in SpinQuant, outlier features only destroy quantization because they are coordinate-aligned along standard basis axes.

By inserting orthogonal rotation matrices $Q \in \mathbb{R}^{d \times d}$ (such as randomized Walsh-Hadamard Transforms) between transformer operations:

$$X_{rotated} = X Q, \quad W_{rotated} = Q^T W$$

Because $Q Q^T = I$, the mathematical result is identical:

$$Y = X_{rotated} W_{rotated} = (X Q)(Q^T W) = X (Q Q^T) W = X W$$

Multiplying by a Walsh-Hadamard matrix redistributes the energy of a 100.0 outlier spike equally across all $d$ dimensions. The maximum activation magnitude drops from $100.0$ to $100.0 / \sqrt{d} \approx 1.56$ for $d = 4096$. The resulting activation tensor becomes strictly Gaussian and outlier-free, allowing straightforward 4-bit (INT4/FP4) quantization across both weights and activations.

Architectural Implications for Modern Model Design

The discovery of emergent outliers has directly shaped subsequent foundation model architectures:

• Query-Key Normalization (QK-Norm): Architectures such as Gemma 2 and Command-R apply RMSNorm directly to query and key vectors prior to dot-product attention ($\text{LN}(Q)\text{LN}(K)^T$), suppressing extreme logit drift and attention sink blowups during pre-training.
• Differential Attention: The Differential Transformer (2024) subtracts two separate softmax attention maps ($A_1 - \lambda A_2$), canceling common-mode activation noise and preventing runaway outlier spikes from dominating the context window.
• Native FP8 Numerical Formats: Modern architectures designed for Hopper and Blackwell GPUs utilize FP8 (E4M3 / E5M2) with fine-grained block scaling (128-element tiles). By confining quantization scale factors to local 1D or 2D blocks, outlier spikes are isolated to a single 128-value vector, preventing dynamic range exhaustion across the broader tensor.

Emergent outlier features represent a fundamental phase change in neural computation: as transformers scale, they transition from distributed continuous representations to specialized coordinate signaling channels. Mastering this phenomenon remains one of the core frontiers bridging mathematical interpretability and production serving efficiency.

Sources

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al., 2022)
• BERT Busters: Outlier Dimensions that Disrupt Transformers (Kovaleva et al., 2021)
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization (Bondarenko et al., 2021)
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (Xiao et al., 2023)
• QuaRot: Outlier-Free 4-Bit Inference in Large Language Models (Ashkboos et al., 2024)
• SpinQuant: LLM Quantization with Learned Rotation (Liu et al., 2024)
• Differential Transformer (Microsoft Research, 2024)