Activation Addition and Representation Engineering: Mathematical Foundations, Linear Subspace Projections, and Inference-Time Steering in Large Language Models

Large language models encode vast linguistic, factual, and behavioral properties within their internal hidden representations. While traditional alignment and behavioral modification rely on Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), these gradient-based techniques modify billions of parameter weights, require substantial compute, and frequently suffer from catastrophic forgetting or alignment tax.

An alternative paradigm grounded in mechanistic interpretability and cognitive neuroscience—known as Activation Addition (ActAdd) or Representation Engineering (RepE)—demonstrates that high-level concepts can be directly monitored and steered at inference time by manipulating intermediate activation vectors without updating model parameters.

Linear subspace projection and contrastive activation geometry

The Linear Representation Hypothesis

Activation steering builds on the Linear Representation Hypothesis, which posits that high-level concepts, task semantics, and behavioral traits are represented as linear directions (one-dimensional rays or low-dimensional subspaces) within the Transformer residual stream.

For a Transformer with $L$ layers and hidden dimension $d$ , the residual stream at layer $l \in \{1, \dots, L\}$ for token position $t$ is represented by the activation vector:

h_l(t) in R^d

Under the linear hypothesis, a semantic attribute or behavioral state $C$ (such as honesty, sycophancy, refusal, sentiment, or reasoning style) corresponds to a unit vector direction $\hat{v}_l \in \mathbb{R}^d$ . The projection of a hidden state onto this direction quantifies the degree to which the model expresses concept $C$ :

score(h_l(t)) = <h_l(t), \hat{v}_l>

When this linear structure holds, modifying the coordinate along $\hat{v}_l$ directly modulates the model's downstream computational trajectory and output behavior.

Concept Direction Extraction Methodologies

Identifying the vector $\hat{v}_l$ that causally mediates a target concept requires reading activations across controlled prompt distributions. Four primary methodologies have been established across the literature:

1. Contrastive Activation Addition (Difference-in-Means)

Introduced by Turner et al. (2023) and expanded by Rimsky et al. (2023), the Difference-in-Means method constructs pairs of contrastive prompts $(x_i^+, x_i^-)$ for $i \in \{1, \dots, N\}$ . The positive prompt $x_i^+$ induces the desired attribute (e.g., "Respond truthfully and objectively: ..."), while the negative prompt $x_i^-$ induces the opposite attribute (e.g., "Respond untruthfully with hallucinations: ...").

The model executes a forward pass on each prompt. For layer $l$ , activations are collected at a designated token position (typically the final prompt token or averaged across the prompt sequence). The steering vector $v_l$ is computed as the empirical mean difference:

v_l = (1 / N) * sum_{i=1}^N ( h_l(x_i^+) - h_l(x_i^-) )

The resulting vector is normalized to unit length:

\hat{v}_l = v_l / ||v_l||_2

2. Linear Artificial Tomography (LAT / PCA)

Developed by Zou et al. (2023) within the Representation Engineering framework, Linear Artificial Tomography constructs a difference matrix $D_l \in \mathbb{R}^{N \times d}$ where each row represents the activation difference for pair $i$ :

d_i = h_l(x_i^+) - h_l(x_i^-)

Principal Component Analysis (PCA) is applied to $D_l$ . The first principal component (the eigenvector associated with the largest eigenvalue of the covariance matrix $D_l^T D_l$ ) captures the dominant axis of variation separating the two behavioral states:

\hat{v}_l = argmax_{||v||=1} v^T (D_l^T D_l) v

This unsupervised projection removes uncorrelated prompt-specific noise while isolating the core concept direction.

3. Contrast Consistent Search (CCS)

Burns et al. (2022) introduced Contrast Consistent Search to discover latent knowledge without relying on ground-truth supervision. For each factual statement $s_i$ , the method forms contrastive pairs with its logical negation $(s_i, \neg s_i)$ .

CCS trains a linear probe $p(h) = \sigma(\theta^T h + b)$ to satisfy fundamental logical consistency constraints:

Probability sum consistency: $p(h(s_i)) + p(h(\neg s_i)) \approx 1$
Informativeness (entropy maximization): The predictions must not collapse to a trivial constant value of $0.5$ .

The optimized probe direction $\theta^*$ defines an unsupervised direction of truthfulness within intermediate representations.

4. Refusal Subspace Isolation

Investigating safety alignment in conversational LLMs, Arditi et al. (2024) demonstrated that refusal behavior across open-weight models is mediated by a single dominant one-dimensional direction. By contrasting activations on harmful requests versus benign instruction-following datasets, the refusal vector $\hat{v}_{refusal}$ isolates the safety gating mechanism across the residual stream.

Inference-Time Intervention Mechanics

Once the concept direction $\hat{v}_l$ is extracted, behavioral control is applied during the inference forward pass without backpropagation or weight updates.

Direct Activation Addition

At layer $l$ , the forward pass computes intermediate activation $h_l(t)$ . Activation addition intervenes by adding a scaled steering vector:

h'_l(t) = h_l(t) + \alpha * \hat{v}_l

Here, $\alpha \in \mathbb{R}$ is the steering coefficient:

$\alpha > 0$ : Amplifies the expression of the target concept.
$\alpha < 0$ : Suppresses the target concept or induces the inverse behavior.
$\alpha = 0$ : Preserves baseline model execution.

The modified activation $h'_l(t)$ passes into subsequent Transformer sublayers (self-attention and feed-forward networks).

Directional Ablation (Abliteration)

To neutralize a specific behavior (such as refusal, specific stylistic quirks, or bias) without actively injecting a strong opposing bias, orthogonal projection removes the concept component entirely from the hidden state:

h'_l(t) = h_l(t) - <h_l(t), \hat{v}_l> * \hat{v}_l

Alternatively, this ablation can be baked permanently into the static weight matrices of the model. For any projection matrix $W$ (such as down-projection in MLPs or attention output matrices $W_O$ ), the weight matrix is orthogonalized against $\hat{v}_l$ :

W' = W - \hat{v}_l * (\hat{v}_l^T * W)

This permanent weight modification achieves inference-time behavioral suppression without requiring custom inference runtime hooks.

Multi-Layer Scheduling and Token Masking

Steering interventions can be applied across single layers, contiguous layer ranges $[l_{start}, l_{end}]$ , or with layer-dependent coefficients $\alpha_l$ .

Furthermore, intervention schedules can target specific token positions:

Prompt-only steering: Intervening on tokens $t \in [1, T_{prompt}]$ conditions the model's interpretation of the input context.
Rollout-only steering: Intervening during autoregressive generation tokens $t > T_{prompt}$ continuously enforces the behavioral constraint across output generation.

Downstream Propagation and Logit Dynamics

To understand why linear modifications in intermediate layers alter token generation probabilities, consider the propagation mechanics through subsequent layers.

In a standard pre-norm Transformer, layer $l$ computes:

a_l = h_{l-1} + Attn(RMSNorm(h_{l-1}))
h_l = a_l + MLP(RMSNorm(a_l))

Because the residual stream operates as an additive communication bus, an intervention $\Delta h_l = \alpha \hat{v}_l$ propagates linearly through the residual connections, while non-linear transformations in subsequent attention heads and MLPs process both the baseline state and the added steering vector:

h_L = h_l + \Delta h_l + sum_{k=l+1}^L ( Attn_k(h_{k-1}) + MLP_k(a_k) )

At the final layer $L$ , un-embedding produces vocabulary logits:

Logits = W_U * RMSNorm(h_L)

The linear component of $\Delta h_l$ that survives the residual path projects directly through the un-embedding matrix $W_U$ :

\Delta Logits \approx \alpha * W_U * \frac{\hat{v}_l}{||h_L||_2}

Tokens whose un-embedding vectors align positively with $\hat{v}_l$ experience an increase in logit values, shifting the output softmax distribution toward concept-aligned vocabulary.

Failure Modes, Critical Thresholds, and Trade-Offs

While activation engineering enables fine-grained control, several operational constraints and failure modes govern its deployment:

1. Off-Manifold Drift and Coherence Collapse

Transformer hidden states reside on a low-dimensional manifold embedded in $\mathbb{R}^d$ , constrained by LayerNorm/RMSNorm regularities. When the steering magnitude $|\alpha|$ exceeds a critical threshold $\alpha_{crit}$ , the modified activation $h'_l$ drifts off the nominal representation manifold.

This produces severe downstream degradation:

Repetition loops and syntax breakdown.
Perplexity spikes on benign tokens.
Complete loss of semantic coherence.

Calibrating $\alpha$ relative to the empirical standard deviation of residual stream activations along $\hat{v}_l$ is essential to maintain syntactic integrity.

2. Semantic Entanglement and Collateral Degradation

Due to polysemanticity and representation compression, linear directions in deep networks are rarely purely orthogonal to all other conceptual representations. Manipulating $\hat{v}_l$ for a single concept often exerts collateral effects on related or un-related capabilities:

Steering for excessive honesty can induce hyper-refusal or factual pedantry that breaks conversational helpfulness.
Suppressing refusal via ablation can inadvertently degrade reasoning on mathematical or coding tasks by disrupting shared feature subspaces.

3. Layer Selectivity Dynamics

Empirical investigations show that steering efficacy varies systematically across network depth:

Early layers (0% to 25% depth): Features are dominated by low-level token identity, positional indexing, and local syntax. Steering here frequently corrupts parsing and token generation.
Middle layers (40% to 75% depth): High-level semantic abstractions, world models, and behavioral policies stabilize. Interventions in these layers yield maximal causal efficacy with minimal syntactic damage.
Late layers (80% to 100% depth): Representations specialize into vocabulary un-embedding preparation and logit calibration. Steering at these layers often acts as a crude token bias rather than semantic modification.

Operational Assessment

Activation addition and representation engineering provide a mathematically transparent, compute-efficient mechanism for analyzing and steering model behavior. By identifying emergent linear directions in activation space, practitioners can implement targeted guardrails, evaluate truthfulness, and customize behavioral styles at runtime without the computational overhead or catastrophic forgetting risks of full parameter fine-tuning.