Mechanistic interpretability.
Neuroscience for neural networks.
Mechanistic interpretability (“mech interp”) is the study of reverse-engineering neural networks to find out exactly how they work inside. Instead of treating a model as a black box that takes an input and returns an answer, it looks for the internal mechanisms that produce the behaviour, and then tests them by intervening. This section explains the field and does some of it: each piece takes a real open language model, states a hypothesis about its internals before looking, and reports what the weights said.
Not what the model does. How it does it.
Ordinary evaluation asks whether a model gets the answer right. Mechanistic interpretability asks which parts of the network computed that answer, in what order, and what each part contributed. The analogy is neuroscience: rather than only observing what a brain does, you trace the circuits and processes that make it happen.
The difference from neuroscience is access. Every weight and every activation of a network can be read, copied, zeroed or swapped, as often as you like, on any input. That is what makes causal claims possible: if you think a component carries a piece of information, you can remove it, or transplant it from another input, and watch whether the output moves.
Three moves, repeated until the story holds.
Look inside
Record the model's internal state on chosen inputs: neuron activations, attention patterns, the residual stream at every layer, the output of every head and MLP.
Find meaningful patterns
Look for components or directions that respond to a concept (names, numbers, syntax, a relation) and stay quiet otherwise. Probes, the logit lens and sparse autoencoders do this at scale.
Reconstruct the mechanism
Trace how information flows from component to component, then prove it by intervention: patch, ablate or steer, and check that the output changes the way the explanation predicts.
“The capital of France is…” Paris. But how?
A model completes the sentence with Paris. An evaluation stops there. A mechanistic interpretability researcher does not stop at the answer being right; they ask four questions, and each one has a technique that answers it.
The goal is a causal explanation of the computation, not a correlation between one neuron and the word “Paris”. A neuron that lights up on “Paris” may be a bystander; a component whose removal turns “Paris” into “Lyon” is not.
- Where, and how, is information about France represented? Probes and the logit lens read the residual stream layer by layer.
- Which components retrieve or process its capital? Attention patterns and per-head attribution narrow it to a few heads and MLPs.
- How does that information move the prediction toward “Paris”? Direct logit attribution splits the final logit into each component's share.
- Can we intervene on those components and change the answer? Activation patching and steering test the explanation by editing the computation.
Five techniques, from one neuron to a whole circuit.
| Technique | What it investigates | Read more |
|---|---|---|
| Neuron analysis | What individual neurons respond to. | The primer |
| Feature analysis | How the model represents concepts, often as directions spread across many neurons. | Superposition |
| Activation patching | Whether changing one internal activation changes the model's behaviour, and by how much. | Patching and causal tracing |
| Circuit analysis | How several components cooperate to perform one computation. | Circuit tracing |
| Sparse autoencoders (SAEs) | Whether dense, tangled activations decompose into a larger set of sparse, more interpretable features. | SAEs |
Four reasons to look inside, and one honest limit.
AI safety
Check whether a model internally represents deceptive strategies, dangerous capabilities or other concerning patterns, rather than trusting its outputs to say so.
Understanding reasoning
See how models carry out multi-step reasoning, use their context, and solve problems, step by step through the layers.
Debugging models
Find why a model makes an error, develops a bias or behaves unexpectedly, and fix the cause instead of the symptom.
Scientific understanding
Discover general principles of how neural networks learn and process information, the way physiology grew out of dissection.
Three questions you can ask of the same model.
Does the model perform well?
Can we explain or summarise why it produced this output?
Which internal computations and components causally produce this output?
The lines are not sharp: the fields borrow each other's tools. What sets mech interp apart is the demand that the explanation be grounded in the model's internal structure and survive an intervention. Attribution methods such as LIME and SHAP explain a model from the outside; the methods here open it.
Hypotheses, tested on real weights.
Each piece is a 3D dissection: an article beside a stage where the model's own numbers are laid out and the camera walks through the computation. Every piece pre-registers its hypotheses and verdict rules before running, reports every seed, and labels anything found after the fact as exploratory.
The first piece is being built. It asks why a model that read “722” in its context sometimes writes “719”: whether a near-tie between two digits is where the slip starts, what the model does after it, and whether checking each token against the context as it is generated can stop it.
Open questions, in the order they will be taken.
The curated list the research routine works through, one piece at a time. Each question is anchored to the paper that raised it, and is answerable on a small model in an afternoon of compute.
-
Open
Number fidelity: why 722 becomes 719
Wu et al. (2024), Retrieval Head Mechanistically Explains Long-Context Factuality, arXiv:2404.15574; Levy & Geva (2024), Language Models Encode Numbers Using Digit Representations in Base 10, arXiv:2410.11781
Question: after the model has written "7", is the slip from 2 to 1 a near-tie between two digits? Once it has written "71", does it complete with the numerically closest number (719) or copy the next source digit (712)? Which heads carry the copy, and does ablating them cause the slip? Can a per-token check against the context stop it without blocking numbers the model is meant to derive?
-
Open
Where the answer forms: logit lens against tuned lens
nostalgebraist (2020), interpreting GPT: the logit lens, LessWrong; Belrose et al. (2023), Eliciting Latent Predictions from Transformers with the Tuned Lens, arXiv:2303.08112
Question: on factual recall, copying and arithmetic prompts, at what depth does the answer become decodable under each lens, and do the two lenses disagree about which layers matter (checked by patching those layers)?
-
Open
Locating a fact: "The capital of France is Paris"
Meng et al. (2022), Locating and Editing Factual Associations in GPT, arXiv:2202.05262; Hase et al. (2023), Does Localization Inform Editing?, arXiv:2301.04213
Question: does the mid-layer MLP peak at the last subject token reproduce at 0.5B to 1.5B, and does editing at the traced layer work better than editing elsewhere, as Hase et al. found it need not?
-
Open
The indirect-object circuit in a modern small model
Wang et al. (2022), Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, arXiv:2211.00593
Question: do the same head classes exist in a model trained a generation later, with grouped-query attention and rotary positions, and how much of the logit difference do they recover?
-
Open
Attention sinks and massive activations
Xiao et al. (2023), Efficient Streaming Language Models with Attention Sinks, arXiv:2309.17453; Sun et al. (2024), Massive Activations in Large Language Models, arXiv:2402.17762
Question: are the sink and the massive activations the same mechanism (remove one, does the other go), and what breaks first when the sink is masked?
-
Open
Sparse autoencoder features on a real residual stream
Cunningham et al. (2023), Sparse Autoencoders Find Highly Interpretable Features in Language Models, arXiv:2309.08600; Bricken et al. (2023), Towards Monosemanticity, Transformer Circuits
Question: how interpretable are the features at this scale, does doubling the dictionary split features as reported, and how much next-token loss does splicing the SAE in cost?
-
Open
A single refusal direction
Arditi et al. (2024), Refusal in Language Models Is Mediated by a Single Direction, arXiv:2406.11717
Question: does one direction mediate refusal in a 0.5B to 1.5B instruct model, at which layer is it cleanest, and what does ablating it cost in general capability?
-
Open
Truth as a direction
Marks & Tegmark (2023), The Geometry of Truth, arXiv:2310.06824
Question: does a truth direction generalise across datasets at this scale, and does adding it flip the model's own true/false answers?
-
Open
Activation steering: ActAdd against contrastive activation addition
Turner et al. (2023), Steering Language Models With Activation Engineering, arXiv:2308.10248; Rimsky et al. (2023), Steering Llama 2 via Contrastive Activation Addition, arXiv:2312.06681
Question: which method gives more behaviour change per unit of fluency lost, and is there a layer band where steering works and outside it nothing does?
-
Open
Detecting hallucination from the inside
Kossen et al. (2024), Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs, arXiv:2406.15927
Question: at 0.5B to 1.5B, does the probe predict semantic entropy and accuracy as well as reported for larger models, and from which layer?
-
Open
Decoding by contrasting layers
Chuang et al. (2023), DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models, arXiv:2309.03883; Shi et al. (2023), Trusting Your Evidence: Hallucinate Less with Context-aware Decoding, arXiv:2305.14739
Question: on the same grounded-generation tasks, which decoder fixes more errors, and do they fix the same ones?
-
Open
Successor heads: incrementing ordinal sequences
Gould et al. (2023), Successor Heads: Recurring, Interpretable Attention Heads In The Wild, arXiv:2312.09230
Question: is there a successor head in a 0.5B model, and does ablating it break "Monday → Tuesday" as well as "3 → 4"?
-
Open
Copy suppression
McDougall et al. (2023), Copy Suppression: Comprehensively Understanding an Attention Head, arXiv:2310.04625
Question: does a copy-suppression head exist in a model with grouped-query attention, and how much does it calibrate the model's confidence?
-
Open
Knowledge neurons, contested
Dai et al. (2022), Knowledge Neurons in Pretrained Transformers, arXiv:2104.08696; Niu et al. (2024), What does the Knowledge Neuron Thesis Have to do with Knowledge?, arXiv:2405.02421
Question: do the attributed neurons control the fact more than random neurons with matched activation, and do they also fire on syntactic patterns, as Niu et al. argue?
-
Open
Function vectors
Todd et al. (2023), Function Vectors in Large Language Models, arXiv:2310.15213
Question: do function vectors exist at 0.5B, which heads carry them, and do they compose?
-
Open
Arithmetic as a bag of heuristics
Nikankin et al. (2024), Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics, arXiv:2410.21272
Question: does a sparse set of heuristic neurons account for addition in a 0.5B model, and do its errors follow the heuristics' gaps?
-
Open
Latent multi-hop reasoning
Yang et al. (2024), Do Large Language Models Latently Perform Multi-Hop Reasoning?, arXiv:2402.16837
Question: how often does the first hop resolve internally at this scale, and is the second hop conditional on it?
-
Open
Finding circuits automatically
Conmy et al. (2023), Towards Automated Circuit Discovery for Mechanistic Interpretability, arXiv:2304.14997; Syed et al. (2023), Attribution Patching Outperforms Automated Circuit Discovery, arXiv:2310.10348
Question: on a task with a known circuit, which method recovers it with fewer edges and less compute, and where do they disagree?
Four ways in, depending on where you start.
The plain-language way
What a network is made of, and why reading it is like reading a brain you can pause.
Neurons, features and circuits
The technical core: why neurons are polysemantic, what a feature is, how heads compose.
How the experiments are done
The causal methods, and the decoding behaviour the first pieces study.
Getting into the research
Where the field publishes, and the open tools and curricula people start with.