Activation Patching and Circuit Discovery in Large Language Models: How Causal Mediation Maps Transformer Subgraphs

Understanding how large language models perform complex reasoning requires moving beyond passive behavioral evaluation. While behavioral benchmarks measure model outputs on specific datasets, they treat the underlying neural network as an inscrutable black box. Mechanistic interpretability aims to reverse-engineer transformer weights and intermediate representations into human-understandable algorithms and computational graphs.

At the core of modern mechanistic interpretability is causal mediation analysis, implemented primarily through activation patching and path patching. By systematically intervening on intermediate activations during inference, researchers can isolate the exact attention heads, multi-layer perceptron (MLP) layers, and information channels that implement specific subroutines, such as factual recall, variable binding, and indirect object identification.

The Limitation of Correlation and Representation Probing

Early interpretability methods relied heavily on linear probing, where a linear classifier is trained on intermediate layer representations to predict specific properties (such as syntax trees or entity attributes). While probing reveals whether information is linearly decodable from a representation, it cannot establish whether the model actually relies on that information to generate its next token. A representation may contain an encoded concept as an epiphenomenon without downstream layers utilizing it during generation.

Establishing true mechanistic causality requires experimental intervention, formalizing Judea Pearl's do-calculus within deep neural networks as demonstrated by Geiger et al. (2021). Rather than observing correlations, researchers execute controlled counterfactual swaps inside the network to measure causal effects directly.

The Mechanics of Activation Patching

Activation patching (also called interchange intervention or causal tracing) operates by running a model on two minimally paired inputs: a clean input $x$ and a corrupted (or counterfactual) input $x^*$.

Consider the canonical indirect object identification (IOI) task studied by Wang et al. (2022):

• Clean prompt $x$: "When Mary and John went to the store, John gave a drink to" $\to$ Expected target: " Mary"
• Corrupted prompt $x^*$: "When Mary and John went to the store, Mary gave a drink to" $\to$ Expected target: " John"

Both sequences share identical grammatical structure, syntactic transitions, and semantic domains, differing only in the subject token that determines the correct recipient.

The activation patching procedure executes three distinct steps:

1. Clean and Corrupted Forward Passes: The model runs on clean prompt $x$, caching all intermediate activation tensors $A(x) = \{a_1, a_2, \dots, a_N\}$ across all attention heads and MLP layers. The model then runs on corrupted prompt $x^$, caching activations $A(x^)$.
2. Intervention Forward Pass: The model runs on the corrupted prompt $x^$, but at a designated component $C$ (such as head $h$ at layer $l$), its activation tensor $a_l^h(x^)$ is replaced with the corresponding clean activation $a_l^h(x)$. The remainder of the network computes normally on the combined state.
3. Metric Evaluation: The causal importance of component $C$ is quantified by measuring the restoration of the clean model output, typically calculated via logit difference:

$$\Delta \text{Logit} = \text{Logit}(\text{Mary}) - \text{Logit}(\text{John})$$

If restoring $a_l^h(x)$ shifts the output distribution from predicting "John" back to predicting "Mary", component $C$ is causally necessary and sufficient within that local computational step to convey the target signal.

A complementary variant, denoising patching (or corrupt-to-clean patching), injects corrupted activations into a clean run to measure performance degradation, isolating components whose disruption breaks the task pipeline.

From Node Patching to Path Patching

Standard activation patching evaluates entire nodes (individual attention heads or MLP blocks). However, node-level patching cannot determine how information flows between components. An attention head performs multiple distinct operations simultaneously: its queries ($Q$) determine where to attend, its keys ($K$) advertise token positions and semantics, its values ($V$) encode the content to be copied, and its output projection ($O$) writes results back to the residual stream.

Path patching, formalized by Wang et al. (2022), resolves this limitation by isolating specific directed edges in the model's computational graph.

Instead of replacing an entire attention head's output across all downstream consumers, path patching intervenes only on the information transmitted along a specific edge from sender component $A$ to receiver component $B$:

1. Sender $A$ computes its activation under the clean input $x$.
2. The network propagates $A(x)$ downstream, but only into the specific input channel of receiver $B$ (e.g., the query vector $Q$ of head $l_{10}h_7$).
3. All other components and input channels of $B$ receive activations derived from corrupted input $x^*$.

Path patching enables researchers to isolate functional sub-circuits, distinguishing whether a head acts as an information retrieval mechanism (modulating attention weights via $Q$ and $K$) or an information transport mechanism (moving token embeddings via $V$ and $O$).

The Computational Bottleneck: Attribution Patching (AtP)

While activation patching provides rigorous causal ground truth, its computational complexity scales poorly with model size.

In an exhaustive edge-level analysis, testing every pair of components across $L$ layers and $H$ heads requires $O(N_{\text{nodes}}^2)$ forward passes. For a small model like GPT-2 (12 layers, 144 heads), full path patching requires thousands of forward passes. For frontier models with 80+ layers and tens of thousands of attention heads, exhaustive activation patching becomes computationally prohibitive.

To overcome this constraint, Syed et al. (2023) and Nanda (2023) developed Attribution Patching (AtP). AtP computes a first-order Taylor series approximation of the activation patching effect using model gradients.

Given a metric $\mathcal{M}(x)$ (such as clean target logit difference) and an activation tensor $a$, the change in metric resulting from patching $a(x^) \to a(x)$ is approximated as: $$\Delta \mathcal{M} \approx (a(x) - a(x^</em>)) \cdot \nabla_a \mathcal{M}(x^*)$$

By caching the activations of the clean and corrupted forward passes and executing a single backward pass on the corrupted input to obtain $\nabla_a \mathcal{M}(x^*)$, Attribution Patching approximates the causal importance of every activation node and edge simultaneously.

While Attribution Patching introduces approximation errors when components exhibit strong non-linearities or multi-head cancellation effects, it achieves orders-of-magnitude speedups, filtering tens of thousands of candidate edges down to a high-priority subset for exact causal validation.

Automated Circuit Discovery: The ACDC Algorithm

Manual circuit discovery requires significant heuristic intervention and expert intuition to design prompts, choose intervention points, and isolate subgraphs. To systematize this workflow, Conmy et al. (2023) introduced Automated Circuit Discovery (ACDC).

ACDC treats the transformer as a directed acyclic graph (DAG) where nodes represent attention heads, MLP layers, and residual stream hooks, and edges represent information flow between them. The algorithm extracts minimal faithful task circuits through automated recursive edge pruning:

1. Initialization: Start with the full computational graph of the model.
2. Reverse Topological Traversal: Iterate backward through edges, from model output logits toward input embeddings.
3. Edge Ablation: For each edge $e = (u, v)$, replace the activation sent from $u$ to $v$ with a corrupted baseline activation (or mean activation).
4. Thresholding: Compute the change in task metric $\mathcal{M}$. If removing edge $e$ changes the metric by less than a pre-defined threshold $\tau$, the edge is permanently pruned from the circuit graph. If the metric degrades significantly, the edge is retained.
5. Circuit Evaluation: The final subgraph is scored on three criteria:

• Faithfulness: How closely the isolated circuit matches the full model's performance on the target task.
• Completeness: Whether any critical computational pathways were omitted.
• Minimality: The sparsity of the circuit (number of active edges relative to the full network).

ACDC demonstrated that algorithmic behaviors across diverse language models can be compressed into compact subgraphs containing less than 5% of the total network parameters.

Case Study: The Indirect Object Identification Circuit

The power of activation patching and circuit analysis is best illustrated by the complete reverse-engineering of the Indirect Object Identification (IOI) circuit in GPT-2 Small by Wang et al. (2022).

Through systematic path patching, the authors identified a sparse circuit of 26 attention heads (out of 144) categorized into distinct functional classes:

1. Duplicate Token Heads (Layers 0 to 3): Detect repeated name tokens in the prompt (e.g., identifying that "John" has appeared twice) and write a signal to the residual stream indicating duplicate status.
2. Induction Heads and Previous Token Heads (Layers 2 to 5): Track sequential token patterns and pass position-relative signals downstream.
3. S-Inhibition Heads (Layers 7 to 9): Read the duplicate token signal from earlier layers and attend specifically to the duplicated subject ("John"). These heads write an inhibitory signal into the query inputs of the downstream Name Mover heads, effectively instructing them not to attend to the duplicate name.
4. Name Mover Heads (Layers 9 to 11): Attend to all candidate names in the prompt context and copy the non-inhibited name ("Mary") directly into the final residual stream, projecting it onto the target vocabulary logit via the unembedding matrix $W_U$.
5. Backup Name Mover Heads: Redundant attention heads that remain largely inactive during standard inference but automatically increase their signal throughput if primary Name Mover heads are ablated (a phenomenon termed the "Hydra effect").

This circuit provided the first empirical proof that transformers do not rely on uniform statistical pattern matching across all layers. Instead, they compose discrete, modular algorithms across specialized attention sub-networks.

Technical Challenges and Frontier Limitations

Despite its successes, activation patching and automated circuit discovery face several structural challenges:

• The Hydra Effect and Self-Repair: Transformers frequently exhibit compensatory redundancy. When an essential head is patched or ablated, downstream heads shift attention patterns in the same forward pass to restore the lost signal, masking the true causal role of the ablated component.
• Polysemanticity and Feature Superposition: Standard activation patching operates on neuron- or head-level nodes. However, individual attention heads and neurons often represent multiple unrelated concepts simultaneously (polysemanticity). Modern research increasingly combines activation patching with Sparse Autoencoders (SAEs) to patch disentangled dictionary features rather than raw architectural nodes.
• Out-of-Distribution Activation Corruption: Replacing an intermediate activation with an arbitrary counterfactual vector can push the downstream network off its natural activation manifold, inducing artificial errors unrelated to the specific task semantics.
• Prompt Sensitivity and Circuit Generalization: A circuit discovered on a synthetic prompt template (such as IOI sentences) often fails to generalize when syntax, vocabulary, or clause ordering varies slightly, requiring broader distributional test harnesses.

Outlook

Activation patching and automated circuit discovery have established mechanistic interpretability as an empirical, hypothesis-driven science. By replacing black-box correlations with rigorous causal interventions, researchers can decompose neural networks into inspectable algorithmic circuits. As automated discovery tools scale to multi-billion parameter foundation models and integrate with sparse feature dictionaries, causal mediation analysis will remain a critical foundation for auditing model reasoning, detecting hidden behavioral misalignment, and verifying AI safety.

Sources

• Towards Automated Circuit Discovery for Mechanistic Interpretability (Conmy et al., NeurIPS 2023)
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (Wang et al., 2022)
• Attribution Patching: Activation Patching At The Speed of Gradients (Syed et al., 2023)
• Locating and Editing Factual Associations in GPT (Meng et al., 2022)
• Causal Distillation for Language Models (Geiger et al., 2021)