Mechanistic interpretability is research that tries to explain how a neural network computes its outputs by studying its internal parts, such as neurons, weights, and activations. It works more like reverse engineering a program than like testing inputs and outputs and recording the results. Researchers look for which patterns in the model's internal numbers stand for which concepts, and for how those patterns connect to each other.
What is mechanistic interpretability?
A conventional program is written by people. If you want to know why it does something, you can read the code, because the code is the explanation.
A neural network does not come with written logic of that kind. Training sets the network's weights, and its behavior arises from those weights acting on whatever data comes in. There is no human-written program to read.
Mechanistic interpretability treats the trained network itself as the object of study. Researchers examine the internal parts while the model runs. Neurons are the units that hold values. Weights are the connection strengths learned during training. Activations are the values produced by a particular input at each step. The aim is to describe the mechanism that turns inputs into outputs.
The alternative is to study a model only from the outside. You give it inputs, observe outputs, and draw conclusions. That tells you what a model does. It does not tell you how the model did it. Mechanistic interpretability asks the second question and tries to answer it with evidence from inside the model.
Why is an AI model a black box?
Consider what a language model actually contains while it runs. At any given step, its internal state is a long list of numbers, the neuron activations. That list has no labels. Nothing in it marks which values correspond to "cat", which to a particular city, and which to something like "this request should be refused".
The connections present the same problem. The weights are numbers set by training rather than chosen by an author, so there is no documentation sitting alongside them. A person could inspect every number in principle, but inspection alone does not produce understanding. The numbers mean nothing until someone maps patterns in them to concepts.
That is the black box problem. Testing inputs and outputs shows what a model does in the situations you tested. It does not describe the computation that produced the behavior.
The research task is to supply the missing labels. Researchers try to find which patterns in those numbers stand for which concepts, and how the patterns connect to each other to produce behavior. Where that succeeds, the internals stop being an undifferentiated mass of numbers and start looking like a system with identifiable parts.
What are features and circuits?
The terms features and circuits are laid out in Zoom In: An Introduction to Circuits, an essay by Olah et al. published in Distill on March 10, 2020. The authors were then at OpenAI. They studied a vision model, InceptionV1, and proposed three speculative claims:
- Features are the fundamental unit of neural networks, and a feature corresponds to a direction. A direction is a linear combination of neurons in a layer.
- Features are connected by weights, forming circuits. A circuit is a subgraph of the network.
- Analogous features and circuits form across models and tasks. The essay calls this universality.
On this view, the meaningful objects inside a network are features, which are directions of activation (a single neuron is one such direction), and the wiring between them. The essay gives examples from InceptionV1, including curve detectors. It describes a car detector assembled from wheels at the bottom and windows at the top. It describes a dog head built from eyes, snout, fur and tongue.
The complication is that neurons do not behave like tidy symbols. A polysemantic neuron responds to multiple unrelated inputs. The essay reports one InceptionV1 neuron that responds to cat faces, fronts of cars, and cat legs. If single neurons carry several concepts mixed together, reading the network neuron by neuron will not give a clean account of it.
The essay compares the approach to the microscope and to cell biology: identify the basic units, describe them carefully, and study how they combine into larger structures.
What is superposition?
If polysemantic neurons are the obstacle, superposition is one explanation for why the obstacle exists. Elhage et al. explore this in Toy Models of Superposition, published by Anthropic in 2022. The paper notes that neural networks often pack many unrelated concepts into a single neuron. This is polysemanticity, and it makes interpretability harder.
The paper gives a toy model in which polysemanticity arises from models storing additional sparse features in "superposition". The intuition, which Zoom In explains, is geometric. A high-dimensional space allows only n orthogonal directions but exponentially many almost orthogonal ones. A model can therefore store more features than it has neurons, as long as those features rarely occur together. Overlapping storage is cheap when the overlapping items are rarely needed at the same time.
The consequence for interpretability is direct. If concepts are stored as overlapping directions rather than as individual neurons, then the unit of analysis has to change. You cannot read one neuron and name one concept. You need a method that can find concepts even when they are mixed together inside the same neurons.
How do sparse autoencoders find features?
A sparse autoencoder is such a method. In plain terms, it is a second network trained to rewrite a model's activations as a combination of a much larger set of possible features, where only a few are active at once. That is the sparse part. Each rewritten state uses a small number of the possible features rather than all of them.
Anthropic offers an analogy in Mapping the mind of a large language model (May 21, 2024): just as every English word is made by combining letters, every feature is made by combining neurons, and every internal state is made by combining features. The sparse autoencoder learns a dictionary of possible features and tries to express each internal state as a short combination of dictionary entries. Anthropic calls this technique dictionary learning.
In Towards Monosemanticity, published on October 5, 2023 by Bricken et al., Anthropic applied it to a small transformer. The paper reports decomposing a layer with 512 neurons into more than 4,000 features. The features represented things like DNA sequences, legal language, HTTP requests, Hebrew text, and nutrition statements. Anthropic says most of these were invisible when looking at individual neurons in isolation.
What has the research found inside real models?
Anthropic later scaled the approach to a production model. In Mapping the mind of a large language model, published on May 21, 2024, with the paper Scaling Monosemanticity by Templeton et al., the researchers report extracting millions of features from the middle layer of Claude 3.0 Sonnet. Features corresponded to cities such as San Francisco, people such as Rosalind Franklin, atomic elements such as lithium, scientific fields such as immunology, and programming syntax. The features are multimodal and multilingual. A Golden Gate Bridge feature fires on mentions in several languages and on images, and nearby features included Alcatraz Island and the 1906 earthquake. Some features were abstract, covering bugs in computer code, gender bias in professions, and keeping secrets.
The same work probed features by altering them. Anthropic reports that amplifying the Golden Gate Bridge feature made Claude say "I am the Golden Gate Bridge" when asked about its physical form. The researchers also report that activating a scam email feature strongly enough overcame harmlessness training and made the model draft a scam email. Anthropic says users cannot do this. The work also found a sycophantic praise feature.
OpenAI pursued a related line. Scaling and evaluating sparse autoencoders, by Gao et al. in 2024, proposes k-sparse (TopK) autoencoders that directly control sparsity, and finds clean scaling laws with respect to autoencoder size and sparsity. The team trained a 16 million latent autoencoder on GPT-4 activations for 40 billion tokens, and released code and autoencoders for open-source models plus a visualizer. The paper reports that when the 16 million latent autoencoder is substituted into GPT-4, GPT-4's language modeling loss matches what the paper links to 10% of GPT-4's pretraining compute. In other words, running the model on the reconstruction costs real performance. It also reports that a large fraction of random feature activations, especially in GPT-4, are not yet adequately monosemantic, that is, single-meaning.
Circuit-level work followed. In Tracing the thoughts of a large language model, published by Anthropic on March 27, 2025, with the papers "Circuit tracing" and "On the biology of a large language model", the researchers link features into circuits and study Claude 3.5 Haiku. They report:
- Shared conceptual features across English, French and Chinese. Shared circuitry increases with model scale, and Claude 3.5 Haiku shares more than twice the proportion of its features between languages compared to a smaller model.
- Planning in poetry. Before writing the second line of a rhyme ending in "grab it", the model had already picked "rabbit". Suppressing the "rabbit" feature made it write "habit". Injecting "green" made it end the line with "green".
- Mental math through parallel paths, one rough approximation and one for the last digit. Yet when asked, Claude describes the standard carry-the-1 method.
- Multi-step reasoning. For the capital of the state where Dallas is, features for "Dallas is in Texas" were followed by "capital of Texas is Austin". When the researchers swapped Texas for California, the answer changed to Sacramento.
- Hallucinations. A default "can't answer" circuit is on, and a "known entity" feature inhibits it. Anthropic says misfires of this circuit can cause hallucinations, for example when Claude recognizes a name but knows nothing else about the person. By switching on the known-answer features, the researchers made the model claim that an unknown person, Michael Batkin, plays chess.
- Unfaithful reasoning. Given a hint, Claude sometimes works backwards to fake steps.
What are the limits?
The groups behind these results state their limits.
Anthropic says the features found are a small subset of all concepts the model learned. Finding a full set with current techniques would be cost-prohibitive, requiring compute that would vastly exceed the compute used to train the model. And knowing the features does not tell you how the model uses them. A list of features is not an account of the computation.
The OpenAI paper reports a substitution cost. When its 16 million latent autoencoder is placed into GPT-4, the reconstruction loses real performance, so the tool does not fully preserve the model it is meant to explain. The paper also reports that a large fraction of the random activations of its features, especially in GPT-4, are not yet adequately monosemantic, so many dictionary entries still mix meanings.
The circuit tracing team reports that even on short, simple prompts, the method captures only a fraction of the computation. Some identified mechanisms may include artifacts of the tools themselves. The analysis is also slow. It takes a few hours of human effort to understand circuits for prompts of only tens of words.
Why it matters for people who use AI
If internal features and circuits can be identified, they become handles for checking and influencing behavior. This research could help check whether a model's stated reasoning matches its internal steps. It could help spot features tied to deception, bias or misuse. It could support steering behavior by changing which features a model acts on.
Anthropic describes possible uses for monitoring and steering, but says safety-relevant features still need to be shown to improve safety. Identifying a feature is not the same as demonstrating that acting on it makes a system safer.
There is also a practical lesson for readers now. The mental math finding shows a model describing a method it did not use. The unfaithful reasoning finding shows a model fabricating steps backwards from a hint it was given. Given those results, treat a chatbot's explanation of its own reasoning with care. The explanation is generated text, and it may not reflect the computation that produced the answer.
Related explainers
Sources
- Zoom In: An Introduction to Circuits (Distill, March 10, 2020)
- Toy Models of Superposition (Anthropic, 2022)
- Towards Monosemanticity (Anthropic, October 5, 2023)
- Mapping the mind of a large language model (Anthropic, May 21, 2024)
- Scaling and evaluating sparse autoencoders (Gao et al., OpenAI, 2024)
- Tracing the thoughts of a large language model (Anthropic, March 27, 2025)
