llms.blog
Interpretability10 articles

Interpretability

Articles

  • Sparse Autoencoders (SAEs) and Mechanistic Interpretability: Mathematical Foundations, Dictionary Learning, Top-K Sparsity, Feature Steering, and Monosemanticity

    Sparse Autoencoders (SAEs) and Mechanistic Interpretability: Mathematical Foundations, Dictionary Learning, Top-K Sparsity, Feature Steering, and Monosemanticity Modern autoregressive large language models represent a vast catalog of world concepts, syntactic rules, and abstract reasoning heuristics. However, inspecting the raw weight matrices and internal activation states of transformer networks reveals an obstinate barrier to mechanistic interpretability: individual neurons are notoriously p

    1 min

  • Activation Addition and Representation Engineering: Mathematical Foundations, Linear Subspace Projections, and Inference-Time Steering in Large Language Models

    Large language models encode vast linguistic, factual, and behavioral properties within their internal hidden representations. While traditional alignment and behavioral modification rely on Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), these gradient-based techniques modify billions of parameter weights, require substantial compute, and frequently suffer from catastrophic forgetting or alignment tax. An alternative paradigm grounded in mechanistic interpret

    1 min

  • Superposition and Sparse Autoencoders: Mathematical Foundations, the Polysemanticity Bottleneck, and Dictionary Learning in Language Models

    For years, attempts to interpret transformer neural networks by inspecting individual neurons encountered an obstinate barrier: polysemanticity. A single neuron in an intermediate multi-layer perceptron (MLP) or residual stream layer rarely corresponds to a single human-interpretable concept. Instead, the same neuron frequently fires on a disparate mixture of inputs, such as Python syntax errors, discussions of Renaissance art, and Spanish verbs. This phenomenon prevents mechanistic interpretabi

    1 min

  • Integrated Gradients: How Axiomatic Attribution Solves the Gradients-at-Saturation Problem in Deep Neural Networks

    Integrated Gradients: How Axiomatic Attribution Solves the Gradients-at-Saturation Problem in Deep Neural Networks Feature attribution methods in deep learning aim to answer a fundamental interpretability question: given an input vector and a trained neural network, how much did each input dimension contribute to the model's final output score? In natural language processing and computer vision, practitioners routinely need to identify which input tokens, pixels, or tabular variables drove a sp

    1 min

  • Centered Kernel Alignment: How CKA Measures Representation Similarity Across Layers and Architectures

    Understanding how deep neural networks represent information across layers, training steps, and disparate architectures has long been a central challenge in machine learning interpretability. When two neural networks are trained on the exact same dataset, even from identical model architectures, their learned weight matrices and individual neuron activations differ completely due to random initialization, data shuffling, and non-convex optimization. Because representations are not aligned to a s

    1 min

  • Superposition and Polysemanticity in Neural Networks: How High-Dimensional Geometry and Sparsity Allow LLMs to Store More Features Than Dimensions

    Superposition and Polysemanticity in Neural Networks: How High-Dimensional Geometry and Sparsity Allow LLMs to Store More Features Than Dimensions In neural network interpretability, one of the most persistent puzzles has been the phenomenon of polysemantic neurons: individual neurons in large language models that activate for seemingly unrelated concepts. A single hidden unit in an MLP layer might fire in response to Python syntax errors, academic citations, and geographic place names. For ye

    1 min

  • Activation Patching and Circuit Discovery in Large Language Models: How Causal Mediation Maps Transformer Subgraphs

    Understanding how large language models perform complex reasoning requires moving beyond passive behavioral evaluation. While behavioral benchmarks measure model outputs on specific datasets, they treat the underlying neural network as an inscrutable black box. Mechanistic interpretability aims to reverse-engineer transformer weights and intermediate representations into human-understandable algorithms and computational graphs. At the core of modern mechanistic interpretability is causal mediat

    1 min

  • The Logit Lens: How Direct Unembedding Decodes Intermediate Representations in Large Language Models

    Autoregressive large language models process input tokens through deep stacks of transformer blocks, generating a final probability distribution over the vocabulary at the final layer. For years, the intermediate computations occurring within these hidden layers were treated as opaque black boxes. Mechanistic interpretability research has demonstrated that modern transformer architectures operate through a structured, iterative refinement process across their residual stream. The logit lens and

    1 min

  • Knowledge Editing in Large Language Models: How Causal Tracing, ROME, and MEMIT Modify Factual Storage in MLP Weights

    Updating factual information in pre-trained large language models has traditionally required two imperfect extremes: computationally expensive continual pre-training, or external prompt-stuffing through Retrieval-Augmented Generation (RAG). Standard gradient descent fine-tuning on isolated facts leads to catastrophic forgetting, parameter drift, and degraded general reasoning. To solve this, mechanistic interpretability researchers introduced direct model editing: a paradigm that treats transfo

    1 min

  • Sparse Autoencoders in Large Language Models: How Dictionary Learning Unpacks Superposition and Neural Monosemanticity

    Deep neural networks have long been treated as uninterpretable black boxes. In transformer language models, individual neurons in the residual stream and multilayer perceptron (MLP) layers rarely map to singular, human-understandable concepts. Instead, individual neurons exhibit polysemanticity: a single neuron might fire for Python syntax, medical terminology, and Korean dialogue without an obvious shared semantic foundation. Mechanistic interpretability research explains this phenomenon throu

    1 min