No Positional Embeddings (NoPE): How Causal Masking and Attention Geometry Encode Sequence Order

A foundational tenet of the Transformer architecture established by Vaswani et al. (2017) is permutation equivariance. Because standard self-attention calculates token interactions purely through pairwise dot products across sets of vectors, shuffling the order of input tokens yields identical outputs up to the corresponding permutation. To establish word order, standard transformer models inject explicit positional information, ranging from learned absolute position embeddings (APE) to sinusoidal encodings, relative position biases (such as T5), and rotary position embeddings (RoPE).

However, research into autoregressive architectures reveals that explicit position embeddings are not strictly necessary for decoder-only models. In architectures with No Positional Embeddings (NoPE), the lower-triangular causal attention mask inherently breaks permutation symmetry. Through causal masking, attention variance shrinkage, and multi-layer residual accumulation, causal transformers naturally construct both absolute and relative positional coordinates from scratch.

The Permutation Asymmetry of Causal Attention

In an unmasked, bidirectional self-attention layer, every token $i$ attends to all $N$ tokens in the sequence. For an input matrix $X \in \mathbb{R}^{N \times d}$, query matrix $Q = X W_Q$, and key matrix $K = X W_K$, the attention weight between token $i$ and token $j$ is:

$$A_{i, j} = \frac{\exp(q_i k_j^T / \sqrt{d_k})}{\sum_{m=1}^N \exp(q_i k_m^T / \sqrt{d_k})}$$

Because the denominator sums across the entire sequence length $N$ regardless of token position, reordering rows in $X$ produces the exact same set of attention logits.

Autoregressive models modify this operation by adding a causal mask matrix $M \in \mathbb{R}^{N \times N}$, where $M_{i, j} = 0$ for $j \le i$ and $M_{i, j} = -\infty$ for $j > i$:

$$A_{i, j} = \frac{\exp((q_i k_j^T / \sqrt{d_k}) + M_{i, j})}{\sum_{m=1}^i \exp(q_i k_m^T / \sqrt{d_k})}$$

This causal constraint fundamentally alters the geometry of the attention operation. The first token ($i = 0$) can only attend to itself, receiving an attention weight of exactly 1.0. The second token ($i = 1$) normalizes its softmax over two positions, the third over three, and the $i$-th token normalizes over $i + 1$ positions. The denominator is no longer invariant to sequence index; it is an explicit function of position $i$.

Attention Variance Shrinkage and Sinks

Empirical work by Haviv et al. (2022) demonstrated that causal transformer language models trained completely without positional encodings match the perplexity of models equipped with standard absolute positional embeddings. Linear probes trained on intermediate hidden states revealed that NoPE representations encode absolute token indices with near 100% accuracy within the first two layers.

Two primary mathematical mechanisms drive this implicit coordinate recovery:

1. Softmax Variance Shrinkage

Under uniform or near-uniform attention distributions, the attention weight allocated to each available preceding token scales inversely with the sequence position:

$$\alpha_{i, j} \approx \frac{1}{i + 1}$$

Consequently, the variance of the aggregated attention output vector $z_i = \sum_{j=0}^i A_{i, j} v_j$ shrinks predictably as position $i$ increases:

$$\text{Var}(z_i) \propto \frac{1}{i + 1}$$

Because the magnitude of the attention output vector varies systematically with the number of attended tokens, feed-forward layers and layer normalization blocks can directly isolate the index $i$ by measuring hidden state norm differentials.

2. Attention Sinks and Anchor Tokens

As documented by Xiao et al. (2023), causal transformers allocate an outsized proportion of attention probability to the initial token in the sequence (the beginning-of-sequence or BOS token), regardless of its semantic relevance. In NoPE models, token 0 serves as a static coordinate reference point.

Because token 0 is always present and receives baseline attention mass, its attention weight $\alpha_{i, 0}$ monotonically decreases as additional tokens are introduced into the denominator sum. The value state $v_0$ effectively acts as an anchor, and the scaling coefficient $\alpha_{i, 0}$ provides a precise signal of the absolute distance $i - 0$.

Constructive Mechanisms Across Layers

Theoretical analysis by Kazemnejad et al. (NeurIPS 2023) provides constructive proofs detailing how multi-layer transformers without positional encodings recover both absolute position and pairwise relative distances.

Layer 1: Absolute Index Accumulation

In the first layer, an attention head can assign uniform positive attention weights across all causally visible tokens by setting $W_Q^{(1)} = 0$ and $W_K^{(1)} = 0$. In this configuration, all valid attention logits equal 0, producing uniform attention weights $A_{i, j} = \frac{1}{i + 1}$ for all $j \le i$.

If the value projection $W_V^{(1)}$ maps a constant bias vector $c$ into the residual stream, the output of the attention head at position $i$ computes:

$$h_i^{(1)} = x_i + \sum_{j=0}^i \frac{1}{i + 1} c = x_i + c$$

By using feed-forward networks (FFNs) to apply non-linear transformations to unnormalized sums, the network can iteratively increment a dedicated counter dimension in the residual stream, yielding a representation that explicitly encodes absolute position $i$.

Layer 2: Computing Pairwise Relative Distance

Once absolute position signals $p_i$ and $p_j$ are embedded into the residual stream at Layer 1, Layer 2 attention heads can compute relative distance $(i - j)$.

Let the Layer 1 hidden state contain an explicit coordinate component: $h_i^{(1)} = [x_i; \beta \cdot i]$. The Layer 2 query and key projections can isolate and multiply these coordinate components:

$$q_i^{(2)} k_j^{(2)T} = (x_i W_Q + \beta i W_{Q, pos})(x_j W_K + \beta j W_{K, pos})^T$$

By configuring the positional sub-matrices such that $W_{Q, pos} W_{K, pos}^T$ acts as an anti-symmetric or offset-measuring bilinear form, the attention logit directly computes linear functions of $(i - j)$. The model reproduces the functionality of explicit relative position bias schemes (such as T5 or ALiBi) entirely through learned weights in standard projection matrices.

Length Generalization and Extrapolation

The choice of positional representation directly determines how well a model generalizes to sequence lengths beyond its training window.

Empirical benchmarks on algorithmic reasoning tasks (including arithmetic addition, string copying, and parity evaluation) show that explicit positional encodings often hinder length extrapolation. Models trained with APE fail immediately beyond their training context because positions $L > L_{train}$ have never received gradient updates. While RoPE and ALiBi preserve relative offsets, their fixed geometric functions can introduce out-of-distribution frequency artifacts when sequence lengths double or quadruple.

In contrast, Kazemnejad et al. (2023) found that NoPE models trained on short sequences extrapolate significantly better to longer sequences on symbolic reasoning benchmarks. Because NoPE learns positional tracking dynamically through causal counting circuits and attention allocation rather than fitting rigid coordinate functions, its internal relative distance mechanisms degrade more smoothly as sequence lengths expand.

Hybrid Architectures: Partial NoPE

While NoPE offers structural simplicity and length extrapolation benefits on algorithmic reasoning, standard language modeling on complex natural text benefits from the inductive bias provided by explicit relative rotations. Pure NoPE models often require more training steps or deeper networks to achieve the same initial pretraining loss on web text as RoPE-equipped models.

To balance pretraining efficiency with length extrapolation, modern architectures frequently adopt Partial NoPE schemes. In a partial rotary configuration with rotation fraction $\rho \in (0, 1)$, rotary transformations are applied only to a subset of head dimensions $d_{rot} = \rho \cdot d_h$, while the remaining dimensions $d_{pass} = (1 - \rho) d_h$ are left unrotated:

$$q_{i} = [R_{\Theta, i} (q_{i, :d_{rot}}); q_{i, d_{rot}:}]$$ $$k_{j} = [R_{\Theta, j} (k_{j, :d_{rot}}); k_{j, d_{rot}:}]$$

This hybrid approach allows the model to leverage explicit rotary geometry for local syntactic dependencies through the rotated subspace, while preserving position-free capacity in the unrotated subspace where causal attention dynamics and soft counting circuits can operate without rigid frequency constraints.

Sources

• Haviv et al. (2022) - Transformer Language Models without Positional Encodings Still Learn Positional Information
• Kazemnejad et al. (2023) - The Impact of Positional Encoding on Length Generalization in Transformers
• Xiao et al. (2023) - Efficient Streaming Language Models with Attention Sinks
• Vaswani et al. (2017) - Attention Is All You Need