In-Context Learning in Large Language Models: How Induction Heads and Attention Circuits Learn Without Weight Updates

Autoregressive large language models demonstrate the ability to adapt to new tasks, follow few-shot demonstrations, and execute algorithmic patterns entirely within their context windows. Unlike traditional fine-tuning, in-context learning occurs at inference time with frozen model parameters, leaving weights completely unchanged ($\Delta \theta = 0$).

For several years following the scaling demonstrations in GPT-3, the internal mechanism governing in-context learning remained an empirical black box. In 2022, mechanistic interpretability researchers at Anthropic identified a fundamental sub-network mechanism responsible for this behavior: induction heads. Detailed by Olsson et al. (2022) and built upon the mathematical circuits framework of Elhage et al. (2021), induction heads represent multi-layer attention circuits that search historical context for previous token transitions and replicate them in current generations.

The Transformer Circuits Framework

To understand induction heads, transformer attention must be analyzed through the lens of mechanistic interpretability rather than isolated matrix multiplications. In standard autoregressive transformers (Vaswani et al., 2017), each attention layer reads from and writes to a shared vector space called the residual stream.

For an input token sequence, the residual stream vector $x_i \in \mathbb{R}^d$ at sequence position $i$ evolves as it passes through successive attention and feedforward layers. An individual attention head $h$ in layer $l$ performs two distinct operations:

1. The Query-Key (QK) Circuit: Determines the attention pattern. It projects the residual stream vectors into Query ($Q$) and Key ($K$) representations via matrices $W_Q \in \mathbb{R}^{d_{head} \times d}$ and $W_K \in \mathbb{R}^{d_{head} \times d}$. The attention weight between destination position $i$ and source position $j$ is:

A_{i,j} = softmax((W_Q x_i)^T (W_K x_j) / sqrt(d_{head}))

1. The Output-Value (OV) Circuit: Determines what content is transferred. It projects the source vector through a Value matrix $W_V \in \mathbb{R}^{d_{head} \times d}$ and down to the residual stream via an Output projection matrix $W_O \in \mathbb{R}^{d \times d_{head}}$:

Output_i = \sum_j A_{i,j} (W_O W_V x_j)

In a single-layer transformer, attention heads are limited to simple direct token lookups, positional offsets, or static bigram statistics. A single-layer model cannot implement conditional sequence continuation because its Query-Key circuit can only compare the current token with past tokens directly. Implementing pattern completion requires compositional attention across multiple layers.

The Minimal Two-Layer Induction Circuit

An induction head is an attention head in layer $L_2$ that implements the abstract pattern:

[A] [B] ... [A] -> predict [B]

To execute this sequence completion rule, the model requires a minimal circuit spanning at least two attention layers ($L_1$ and $L_2$), operating as a coordinated team:

Step 1: The Previous-Token Head (Layer 1)

In the first attention layer, a specialized attention head (known as a Previous-Token Head) attends strictly to position $i - 1$. When reading token $B$ at position $i$, this head attends to token $A$ at position $i - 1$. Through its Output-Value circuit, it writes information about token $A$ directly into the residual stream at position $i$.

Consequently, after Layer 1, the residual stream vector at position $i$ carries two distinct pieces of information:

• The current token identity: $B$
• The predecessor token identity: "My predecessor was $A$"

Step 2: The Induction Head (Layer 2)

In the second attention layer, the Induction Head executes the matching operation. When the model reaches a subsequent occurrence of token $A$ at destination position $j$:

• Query Projection ($W_Q$): The induction head at position $j$ reads token $A$ and produces a query vector representing: "Search for tokens whose predecessor was $A$."
• Key Projection ($W_K$): At historical position $i$, the key projection reads the Layer 1 output stored in token $B$'s residual stream ("My predecessor was $A$").
• Attention Score: The dot product $(W_Q x_j)^T (W_K x_i)$ produces a large positive score, causing position $j$ to place near-total attention weight on position $i$ (the token $B$).
• Value-Output Projection ($W_O W_V$): The induction head extracts token $B$'s content from position $i$ and writes it into position $j$'s residual stream, steering the language model output head to predict $B$ as the next token.

This composition allows the transformer to detect repeated sequences of arbitrary length and copy the appropriate successor token without any gradient parameter updates.

Emergence and the Macroscopic Phase Change

During the pre-training of autoregressive language models, induction heads do not develop continuously or linearly. Instead, research by Olsson et al. (2022) revealed that induction heads form abruptly in a sharp phase change early in training.

Training Steps
0% --------------> 2.5% [Phase Transition] ---------------------> 100%
                      |
                      +-- Induction Head Formation Spike
                      +-- Sharp Drop in In-Context Loss
                      +-- Sudden Few-Shot Capability

This phase change exhibits several distinct empirical markers across model sizes (from small toy transformers to multi-billion parameter foundation models):

1. Formation of Induction Circuits: Scores measuring induction behavior across attention heads spike simultaneously within a narrow band of training tokens.
2. In-Context Loss Divergence: The per-token cross-entropy loss on tokens later in a context window (e.g., tokens 500 to 2000) drops dramatically compared to early tokens (tokens 1 to 50), demonstrating that the network has learned to exploit long prefixes.
3. Few-Shot Task Performance: Zero-shot and few-shot prompt evaluation curves experience their steepest inflection point at the exact training step where induction heads crystallize.

When researchers experimentally ablated or knocked out induction heads in trained models, the models' ability to perform in-context few-shot learning deteriorated significantly, confirming that these circuits are causal drivers of in-context adaptation.

Generalized and Semantic Induction

While the canonical induction head performs exact token copying ($[A][B] \dots [A] \rightarrow [B]$), real-world transformer workloads require higher-level abstractions. Extended interpretability research has shown that modern models develop generalized variations of induction circuits:

• Prefix and N-Gram Matching: Multi-head clusters that match variable-length sequences ($[A][B][C] \dots [A][B] \rightarrow [C]$), suppressing spurious single-token matches.
• Translation and Cross-Lingual Induction: Circuits that map concepts across representations, such as matching English demonstration pairs ($[Word_{en}] [Word_{fr}] \dots [Query_{en}] \rightarrow [Query_{fr}]$).
• Semantic Induction Heads: Research by Ren et al. (2024) demonstrated that later transformer layers host semantic induction heads. Rather than requiring exact lexical matches, these heads activate on synonymous concepts, syntactic roles, or shared ontological categories, enabling abstract analogy completion.
• Variable Binding in Code: In programming tasks, induction circuits track variable declarations and their assigned types or values, ensuring consistent identifier usage across long functions.

Theoretical Framing: In-Context Learning as Implicit Optimization

The discovery of induction heads bridges mechanistic circuit analysis with theoretical machine learning perspectives. Several theoretical studies, including von Oswald et al. (2023) and Dai et al. (2023), demonstrated that linear attention layers can mathematically simulate steps of gradient descent (meta-optimization).

In this view, the forward pass of a transformer implements an optimization algorithm where:

• The activation state acts as internal dynamic parameters.
• The attention mechanism computes implicit error gradients from few-shot examples.
• Induction circuits serve as the low-level data-routing primitives that retrieve, compare, and update state representations according to observed demonstrations.

Operational Constraints and Long-Context Degradation

Understanding induction heads highlights specific structural constraints in modern LLM architectures:

• Positional Encoding Sensitivity: Induction circuits depend on accurate relative distance representations. Architectures utilizing Rotary Position Embeddings (RoPE) maintain relative token offsets more robustly than absolute position embeddings, facilitating induction head formation over extended context lengths.
• Retrieval Dilution over Long Contexts: As context windows scale to 128k or 1M tokens, the softmax attention distribution over thousands of keys can suffer from attention dilution, where the signal from a single past predecessor token is drowned out by background noise unless reinforced by strong positional decay or fine-tuned retrieval heads.
• Layer Depth Requirements: Because a minimal induction circuit requires at least two attention layers, compact single-layer or shallow student models in distillation pipelines cannot natively form induction circuits, limiting their autonomous few-shot learning capacity.

Induction heads demonstrate that complex emergent behaviors in large language models often stem from discrete, modular algorithmic circuits embedded directly in the network weights.

Sources

• In-context Learning and Induction Heads (Olsson et al., 2022)
• A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)
• Language Models are Few-Shot Learners (Brown et al., 2020)
• Attention Is All You Need (Vaswani et al., 2017)
• Identifying Semantic Induction Heads to Understand In-Context Learning (Ren et al., 2024)
• Transformers Learn In-Context by Gradient Descent (von Oswald et al., 2023)
• Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers (Dai et al., 2023)
• RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)