The QK and OV Circuits in Transformers: How Bilinear Attention Routing and Subspace Projections Move Information

The standard mathematical presentation of multi-head self-attention, introduced in Vaswani et al. (2017), describes the layer as a sequence of matrix projections followed by scaled dot-product operations, concatenation, and an output projection. While computationally efficient for parallel GPU hardware, this formulation obscures the fundamental linear mechanics governing how transformers process, route, and transform representations.

Through the mechanistic interpretability framework developed in Elhage et al. (2021), multi-head attention can be refactored into two mathematically independent, low-rank operations per head: the Query-Key (QK) circuit and the Output-Value (OV) circuit. The QK circuit computes a bilinear form that dictates where information moves across token positions, while the OV circuit executes a linear transformation that dictates what information is transferred into the residual stream.

Understanding attention through the lens of QK and OV factorization clarifies how transformer layers interact additively, how multi-layer composition gives rise to in-context learning algorithms such as induction heads (Olsson et al., 2022), and how specific attention heads implement targeted linguistic and algorithmic functions.

1. Deconstructing Attention into Independent Linear Operations

In standard implementations, given an input representation matrix X across N token positions in a d_model dimensional space, an attention layer with H heads computes queries, keys, and values using projection matrices W_Q, W_K, and W_V. The outputs across all heads are concatenated and multiplied by an output projection W_O.

As demonstrated by Elhage et al. (2021), the output projection W_O can be partitioned into head-specific blocks [W_O^1, W_O^2, ..., W_O^H], where each block maps directly from the head dimension d_head back to d_model. The concatenated matrix multiplication is algebraically identical to running each head independently and summing their contributions into the residual stream:

Layer Output = sum_{h=1}^H (W_O^h * r^h)

Where r^h represents the aggregated value vectors for head h across sequence positions.

When examined on an individual head basis, attention performs two distinct, decoupled operations:

• Information Routing (The QK Circuit): Computing an attention pattern matrix A based on token representations.
• Representation Transformation (The OV Circuit): Applying a linear map W_OV = W_V * W_O that reads from a source token's residual stream and writes to the destination token's residual stream.

Using tensor product notation (where (x) denotes the Kronecker product), the output of head h applied to the input matrix X is expressed as:

Head^h(X) = (A^h (x) W_OV^h) * X = A^h * X * W_V^h * W_O^h

In this formulation, A^h mixes representations across the sequence dimension (positions), while W_OV^h operates independently across the feature dimension (subspaces of the residual stream).

   Source Residual Stream (Position j)
                 │
                 ▼
        ┌─────────────────┐
        │  W_V (d_v x d)  │ <── Read from Source Subspace
        └────────┬────────┘
                 │ (d_head bottleneck)
                 ▼
        ┌─────────────────┐
        │  W_O (d x d_v)  │ ──> Write to Destination Subspace
        └────────┬────────┘
                 │
                 ▼
  Scaled by Attention Weight A[i, j] = softmax(x_i^T W_QK x_j / sqrt(d_k))
                 │
                 ▼
Destination Residual Stream (Position i)

2. The QK Circuit: Bilinear Forms and Token Addressing

The Query-Key circuit governs token addressing: determining which source positions j provide information to a given destination position i.

In conventional formulations, query vectors q_i = x_i * W_Q and key vectors k_j = x_j * W_K are computed separately, and their inner product is evaluated. However, because W_Q and W_K only ever interact via the inner product, they can be merged into a single low-rank matrix W_QK:

W_QK = W_Q * (W_K)^T

The pre-softmax attention score alpha_{i,j} between destination token x_i and source token x_j is given by the bilinear form:

alpha_{i,j} = (x_i * W_Q * (x_j * W_K)^T) / sqrt(d_head)
            = (x_i * W_QK * (x_j)^T) / sqrt(d_head)

The resulting attention pattern matrix A is obtained by applying the causal row-wise softmax:

A = softmax( (X * W_QK * X^T) / sqrt(d_head) + M )

Where M is the causal autoregressive mask (M_{i,j} = 0 for j <= i, and -infinity for j > i).

Key Mathematical Properties of the QK Matrix

• Rank Bottleneck: Because W_Q and W_K have inner dimension d_head, the matrix W_QK has a maximum rank of d_head (typically 64 or 128), whereas d_model ranges from 768 in small models to 8,192 or more in frontier architectures.
• Bilinear Matching: W_QK defines an asymmetric bilinear metric. The symmetric component (W_QK + W_QK^T) / 2 scores feature similarity between tokens, while the asymmetric component (W_QK - W_QK^T) / 2 encodes directional semantic relationships, such as verb-to-object or modifier-to-noun bindings.
• Gauge Invariance: Intermediate vectors q_i and k_j are not unique. For any invertible matrix R of dimension d_head x d_head, setting W_Q' = W_Q * R and W_K' = W_K * (R^-1)^T leaves W_QK unchanged. Consequently, mechanistic analysis focuses on the invariant operator W_QK rather than isolated key and query spaces.

3. The OV Circuit: Linear Maps and Subspace Transformations

While the QK circuit determines the scalar routing weights A_{i,j}, the Output-Value circuit dictates the semantic content transferred from position j to position i.

The matrix W_OV is defined as:

W_OV = W_V * W_O

When head h attends from position i to position j, the contribution added to the destination residual stream x_i is proportional to:

Delta x_i = A_{i,j} * (x_j * W_OV)

Singular Value Decomposition of the OV Circuit

Like W_QK, the matrix W_OV is rank-bounded by d_head. Its internal mechanics can be fully characterized by its Singular Value Decomposition (SVD):

W_OV = U * Sigma * V^T = sum_{k=1}^{d_head} sigma_k * u_k * (v_k)^T

Where:

• The right singular vectors v_k define the reading subspace: the specific linear combinations of features in the source residual stream that the head is sensitive to.
• The left singular vectors u_k define the writing subspace: the directions in the destination residual stream where transformed information is deposited.
• The singular values sigma_k represent the operational gain for each feature channel.

Because d_head is much smaller than d_model, attention heads read from and write to highly restricted linear subspaces. This allows multiple attention heads in the same layer to operate in parallel without destructive interference, reading from distinct source subspaces and writing to orthogonal destination subspaces in the shared residual stream.

Residual Stream (High Dimensional Space, d_model)
┌───────────────────────────────────────────────────────────────┐
│                                                               │
│   Source Subspace V_k ──> [ W_OV Projection ] ──> Dest U_k    │
│   (Read by Head h1)                               (Written)   │
│                                                               │
│   Source Subspace V_m ──> [ W_OV Projection ] ──> Dest U_m    │
│   (Read by Head h2)                               (Written)   │
│                                                               │
└───────────────────────────────────────────────────────────────┘

4. End-to-End Logit Paths in 1-Layer Attention Models

To understand how QK and OV circuits combine to produce model outputs, consider a 1-layer attention-only transformer. The full computation from input token sequence to output vocabulary logits can be expanded into explicit, interpretable paths.

Let W_E be the token embedding matrix, W_pos be the positional embedding matrix, and W_U be the unembedding matrix.

The output logits L across sequence positions can be decomposed as:

L = (X + sum_{h=1}^H Head^h(X)) * W_U 
  = X * W_U + sum_{h=1}^H (A^h * X * W_OV^h * W_U)

Substituting X = E + P (where E represents token embeddings and P represents positional embeddings):

L = E * W_U + P * W_U + sum_{h=1}^H (A^h * E * W_OV^h * W_U) + sum_{h=1}^H (A^h * P * W_OV^h * W_U)

Path Decomposition Breakdown

• **The Direct Token Path (E * W_U):** This matrix represents the static bigram log-likelihood. Without moving information across positions, the model predicts the subsequent token based solely on the identity of the current token (for example, predicting "York" immediately after "New").
• **The Full OV Logit Map (W_E * W_OV^h * W_U):** This matrix maps source token vocabulary items directly to destination token logit updates. If entry (u, v) is large and positive, attending to token u actively increases the output logit for token v.
• **The Full QK Vocabulary Map (W_E * W_QK^h * (W_E)^T):** This matrix dictates token-to-token semantic attention preferences. Entry (u, v) measures the affinity of destination token u for source token v, independent of position.

Together, the combination of W_E * W_QK^h * (W_E)^T and W_E * W_OV^h * W_U allows 1-layer attention heads to act as skip-trigram models: if token A occurs at position j and token B occurs at position i, the QK circuit triggers attention from i to j, and the OV circuit writes logits promoting token C, effectively implementing rules of the form "[A] ... [B] -> [C]".

5. Circuit Composition Across Layers

In architectures with two or more layers, attention heads do not merely process static input embeddings. Instead, heads in layer l+1 read residual stream representations that have already been modified by heads in layer l.

As detailed in Elhage et al. (2021), there are three distinct pathways by which an earlier head h1 in Layer 1 can compose with a later head h2 in Layer 2:

• **Q-Composition (W_OV^{h1} * W_Q^{h2}):** Head h1 modifies the destination token representation, altering what queries Head h2 generates and where it searches in the sequence.
• **K-Composition (W_OV^{h1} * W_K^{h2}):** Head h1 modifies the source token representation, altering the key features that Head h2 matches against.
• **V-Composition (W_OV^{h1} * W_V^{h2}):** Head h1 modifies the information payload, which Head h2 reads, transforms, and passes forward into downstream layers.

Layer 1 Head (h1)                   Layer 2 Head (h2)
┌─────────────────┐                 ┌─────────────────┐
│     W_OV^h1     │ ────> K-Comp ──>│     W_K^h2      │ ──> Sets Attention Pattern A^h2
│ (Writes feature)│                 └─────────────────┘
│                 │ ────> V-Comp ──>┌─────────────────┐
│                 │                 │     W_V^h2      │ ──> Transforms Payload to W_O^h2
└─────────────────┘                 └─────────────────┘

The Mechanistic Architecture of Induction Heads

The most prominent example of circuit composition is the induction head, identified by Olsson et al. (2022) as the primary engine of in-context few-shot learning across language models.

An induction head detects repeated sequence patterns of the form [A][B] ... [A] and completes the pattern by predicting [B]. It requires a two-head composition across layers:

1. Previous-Token Head (Layer 1): A Layer 1 head uses its QK circuit to attend to position i-1 from position i. Its OV circuit copies the identity of token i-1 into the residual stream at position i.
2. Induction Head (Layer 2): A Layer 2 head uses K-composition (W_OV^{L1} * W_K^{L2}). Its query vector at position n reads the current token A. Its key vector at position j reads the output of the Layer 1 head, which contains the token from position j-1.
3. Pattern Trigger: When the query for token A matches the key (which encodes that the previous token was also A), the QK circuit attends to position j.
4. Logit Prediction: The OV circuit of the Layer 2 head (W_OV^{L2}) reads the token at position j (which is B) and writes positive logits for token B directly into the unembedding path via W_U.

Without the decomposition into QK and OV circuits, the emergence of this algorithmic circuit across non-adjacent layers would appear as an opaque, distributed activation pattern.

6. Spectral Analysis of Circuit Matrices

Analyzing the eigenvalue spectra of W_OV and W_QK reveals functional specializations across individual attention heads:

OV Eigenvalues and Information Copying

Because W_OV maps a vector space back into itself (from d_model to d_model), its eigenvalues lambda_k provide direct insight into its behavior:

• Positive Real Eigenvalues: Heads with predominantly positive real eigenvalues act as copying heads. When attending to a token, they reinforce the features of that token in the residual stream without inversion or dimensional rotation.
• Negative Real Eigenvalues: Heads with negative real eigenvalues act as information erasers or suppressors. In mechanistic interpretability audits (Wang et al., 2022), these heads appear as backup or negative heads that conditionally subtract over-represented logits to stabilize prediction calibration.
• Complex Eigenvalue Pairs: Heads with substantial imaginary components apply rotational transformations to representation vectors, encoding syntactic shifts or relational mappings between source and destination subspaces.

7. Practical Applications in Model Engineering and Interpretability

Refactoring attention into QK and OV circuits provides concrete engineering tools for inspecting and optimizing production models:

1. Direct Logit Attribution (DLA): By calculating x_j * W_OV^h * W_U, engineers can determine the exact contribution of any individual attention head to the final token prediction without performing full forward-backward sweeps or ablation testing.
2. Circuit-Level Head Pruning: Standard magnitude-based weight pruning frequently damages critical compositional circuits. By computing the Frobenius norm and spectral rank of W_QK and W_OV, heads with negligible transmission gain across active subspaces can be pruned or merged without degrading in-context reasoning performance (Michel et al., 2019).
3. Targeted Model Editing and Safety Interventions: Localization techniques such as ROME (Meng et al., 2022) and causal mediation analysis identify whether factual recall is driven by MLP associative memory or attention-mediated OV subspace copying, allowing targeted surgical intervention on misaligned behaviors.

Sources

• Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., & Olah, C. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html
• Olsson, C., Elhage, N., Nanda, N., Joseph, N., Nova, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., & Olah, C. (2022). In-context Learning and Induction Heads. Transformer Circuits Thread. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html
• Vaswani, A., Shazeer, N., Parmar, J., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS 2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
• Wang, K., Variengien, A., Conmy, A., Shlegeris, B., & Steinhardt, J. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. arXiv preprint arXiv:2211.00593. https://arxiv.org/abs/2211.00593
• Michel, P., Levy, O., & Neubig, G. (2019). Are Sixteen Heads Really Better Than One?. Advances in Neural Information Processing Systems (NeurIPS 2019). https://arxiv.org/abs/1905.10650
• Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems (NeurIPS 2022). https://arxiv.org/abs/2202.05262