Attention with Linear Biases (ALiBi): How Static Positional Slopes Enable Zero-Shot Context Extrapolation

Large language models process sequences by transforming discrete tokens into continuous vector representations. Standard dot-product self-attention is permutation-invariant: without explicit positional information, the attention operation treats a sequence as an unordered bag of tokens. Early transformer architectures addressed this limitation using Absolute Positional Embeddings (APE), either through fixed sinusoidal functions or learned lookup tables added directly to token embeddings.

While absolute embeddings allow models to distinguish token ordering within their training window, they fail when exposed to longer context lengths at inference. Models trained with sinusoidal or learned embeddings experience severe perplexity degradation when sequence length exceeds the pre-training context window $L_{train}$ .

Attention with Linear Biases (ALiBi), introduced by Ofir Press, Noah A. Smith, and Mike Lewis at ICLR 2022, resolved this limitation by removing positional embeddings from token representations entirely. Instead, ALiBi injects a static, non-learned linear penalty directly into the query-key attention score matrix, enabling zero-shot context length extrapolation.

The Mathematical Mechanics of ALiBi

In standard multi-head attention, the raw attention score between query token $i$ and key token $j$ is computed as:

$\text{Score}(i, j) = \frac{\mathbf{q}_i \mathbf{k}_j^\top}{\sqrt{d_k}}$

ALiBi modifies this operation by subtracting a distance penalty scaled by a fixed, head-specific slope $m$ :

$\text{Score}_{\text{ALiBi}}(i, j) = \frac{\mathbf{q}_i \mathbf{k}_j^\top}{\sqrt{d_k}} - m \cdot (i - j)$

Where:

$i$ represents the query position index.
$j$ represents the key position index (with $j \le i$ in causal auto-regressive decoding).
$(i - j)$ is the non-negative token distance.
$m$ is a static, head-specific scalar slope that does not update during backpropagation.

After applying the causal attention mask and softmax normalization, the resulting attention weights determine how much probability mass query $i$ allocates to key $j$ .

Query i  Key j (Distance: i - j)
  [0]      [0]   -> Penalty: 0 * m
  [1]      [0]   -> Penalty: 1 * m
  [2]      [0]   -> Penalty: 2 * m
  [N]      [0]   -> Penalty: N * m  (Monotonically decays attention)

Because the penalty grows linearly with token distance, attention scores naturally decay for tokens situated further back in the sequence.

Multi-Head Attention Slopes and Positional Penalty Decay

Geometric Slope Distribution Across Attention Heads

A single scalar slope cannot accommodate both short-range syntax tracking and long-range semantic retrieval. To provide multi-scale temporal resolution, ALiBi distributes slopes across the attention heads of each layer using a fixed geometric sequence.

For a transformer layer containing $H$ attention heads (where $H$ is a power of 2), the slope $m_h$ for head $h \in \{1, 2, \dots, H\}$ is defined as:

$m_h = 2^{-\frac{8 \cdot h}{H}}$

For an 8-head model ( $H = 8$ ), the geometric ratio is $2^{-8/8} = 2^{-1} = 0.5$ . The resulting slopes evaluate to:

$m \in \left\{ \frac{1}{2^1}, \frac{1}{2^2}, \frac{1}{2^3}, \frac{1}{2^4}, \frac{1}{2^5}, \frac{1}{2^6}, \frac{1}{2^7}, \frac{1}{2^8} \right\} = \{0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, 0.00390625\}$

When the number of heads $H$ is not a power of 2, the algorithm determines the nearest power of 2 below $H$ , generates the base geometric sequence, and interpolates additional slopes from a higher-frequency sequence to maintain balance.

This mathematical configuration yields structural specialization across the model:

Steep Slopes (e.g., $m = 0.5$ ): Penalize distance heavily. The attention score drops by 0.5 logits per token separation. At a distance of 20 tokens, the logit penalty reaches $-10$ , driving the post-softmax attention weight near zero. These heads function as localized syntactic collectors.
Shallow Slopes (e.g., $m = 0.0039$ ): Impose minimal distance attenuation. At a distance of 1,000 tokens, the logit penalty is only $-3.9$ , allowing content-driven semantic matching to dominate over token distance. These heads preserve long-range retrieval.

The Train-Short, Test-Long Property

Prior to ALiBi, extending context windows required training on full target sequence lengths. Training on long sequences incurs quadratic computational and memory costs: doubling sequence length quadruples attention computation and doubles KV cache footprint.

The original empirical evaluation in Press et al. (2021) demonstrated that models trained with ALiBi on sequence lengths of 1,024 tokens could extrapolate to 2,048 tokens zero-shot, matching the perplexity of baseline sinusoidal models trained natively on 2,048 tokens. This setup yielded an 11% reduction in training time and an 11% reduction in peak memory consumption while maintaining stable perplexity on validation sequences up to 16,384 tokens.

In contrast, models utilizing Sinusoidal Position Embeddings, Rotary Position Embeddings (RoPE), or learned absolute embeddings suffered catastrophic perplexity explosions when evaluated at sequence lengths beyond $1.2\times$ their training budget.

Architectural Comparison: ALiBi vs. RoPE and NoPE

Positional representations in modern autoregressive transformers divide into four distinct paradigms:

Absolute Positional Embeddings (APE / Sinusoidal): Injected directly into token embeddings ( $\mathbf{x}_i + \mathbf{p}_i$ ). Employs global coordinate lookup tables or fixed sinusoidal trigonometric series. Fails zero-shot extrapolation beyond $1.2\times L_{train}$ because query-key dot products drift into unseen coordinate regions.
Rotary Position Embeddings (RoPE): Multiplied directly onto query and key projections ( $\mathbf{R}_{\Theta, i}^d \mathbf{q}_i$ ). Preserves relative distance via complex rotation in two-dimensional subspaces. Requires post-hoc frequency interpolation methods (such as YaRN or NTK-aware scaling) to handle extended context windows without phase drift.
Attention with Linear Biases (ALiBi): Injected as an additive scalar bias directly into attention logits ( $\mathbf{q}_i \mathbf{k}_j^\top / \sqrt{d} - m(i-j)$ ). Requires no learnable parameters and provides native zero-shot length extrapolation because attention logits are strictly bounded by monotonic linear decay.
No Position Embeddings (NoPE): Omits positional encoding entirely. Relies solely on causal masking asymmetry for ordering cues. While partially robust to sequence scaling, it degrades significantly on complex structural and permutation tasks.

While RoFormer's Rotary Position Embedding rotates the query and key vectors in two-dimensional slices according to position $i$ , it alters the geometric angles between representations. When evaluated at lengths $L > L_{train}$ , unseen rotation angles push the query-key inner products into uncalibrated regimes. RoPE therefore requires specialized context extension methods such as Position Interpolation or YaRN.

ALiBi avoids out-of-distribution logit values because adding negative distances simply pushes long-range attention weights toward zero, ensuring stable softmax output distributions.

Practical Limitations and Serving Trade-Offs

Despite its theoretical advantages for length extrapolation, ALiBi introduces specific architectural trade-offs:

1. The Fog-of-War Effect

Because the penalty $-m \cdot (i - j)$ is strictly monotonic, ALiBi assumes that older tokens are intrinsically less relevant than recent tokens. In tasks requiring precise needle-in-a-haystack retrieval from early context (such as referencing system instructions or initial variable declarations placed 32k tokens in the past), even the shallowest slopes introduce substantial negative biases (e.g., $-0.0039 \times 32{,}000 = -124.8$ ). This logit suppression can effectively blind the attention mechanism to distant critical tokens.

2. Numerical Precision Collapses in Low-Precision Floats

In standard FP16 or BF16 representations, the dynamic range and mantissa resolution are constrained. As documented in SambaNova's positional encoding analysis, at context lengths exceeding 8,000 tokens, the linear subtraction can suffer from step discretization. Differences between consecutive positions $(i - j)$ and $(i - j + 1)$ drop below machine epsilon for specific exponent ranges, causing multiple adjacent tokens to receive identical positional penalties.

3. Incompatibility with FlashAttention-1 Optimizations

Early fused attention implementations like FlashAttention-1 were designed around standard matrix multiplication kernels without arbitrary per-element additive bias tensors. Injecting dynamic bias matrices of shape $(B, H, S, S)$ created memory bandwidth overheads unless custom fused CUDA kernels (such as Triton or FlashAttention-2 with ALiBi support) were explicitly deployed.

Production Adoptions and Legacy

ALiBi has served as the positional foundation for several notable open-weight language models:

BLOOM (176B): The BigScience consortium selected ALiBi for its flagship 176-billion-parameter multilingual model, as detailed in the BLOOM architectural technical report, prioritizing multi-scale stability across 46 natural languages and 13 programming languages.
MPT-7B and MPT-30B: MosaicML utilized ALiBi in its MPT series, successfully extrapolating models trained on 2,048 tokens up to 65k and 84k context windows during inference without fine-tuning.
Falcon and Baichuan: Several early multi-billion parameter foundation architectures deployed ALiBi or hybrid variations to facilitate inference scaling without retraining.

While contemporary frontier models (such as Llama 3, Mistral, and Qwen 2.5) have largely coalesced around high-base RoPE combined with YaRN-style temperature scaling due to superior associative recall across deep context, ALiBi established the empirical foundation for relative positional inductive biases in modern transformer engineering.