Activation Patching and Causal Tracing
Swap one internal activation from a corrupted run into a clean run and measure how much of the output moves; done systematically it localises where in a network a specific computation happens.
Ablation answers "does the model still work without this component". That is the wrong question, because a component can be necessary without being where the interesting work happens, and models are redundant enough that removing one head often changes nothing.
Activation patching asks a sharper question: run the model on a clean prompt and a corrupted one, then copy a single internal activation from one run into the other and see how far the output moves. What moves is causally downstream of that activation, on this input, for this behaviour.
The recipe
Take a prompt pair differing in one semantic feature. Clean: "The Eiffel Tower is in" (target: Paris). Corrupted: "The Colosseum is in" (target: Rome).
- Run clean, cache every activation.
- Run corrupted, cache every activation.
- For each site (a layer's residual stream at a token, an attention head's output, an MLP output), rerun the corrupted forward pass with that one site replaced by its clean value.
- Score the recovery, typically the logit difference between the two targets, restored as a fraction of the clean-to-corrupted gap.
Sweeping over layers and token positions gives a heat map, and its bright regions are where the information that distinguishes the two prompts is carried.
Meng et al. used exactly this under the name causal tracing and found that factual recall is mediated by a distinct set of steps in mid-layer feed-forward modules at the subject token, then built ROME to edit those weights directly (Locating and Editing Factual Associations in GPT, NeurIPS 2022, arXiv:2202.05262). Wang et al. used patching plus path patching to isolate a 26-head, 7-group circuit for indirect object identification in GPT-2 small (Interpretability in the Wild, arXiv:2211.00593).
Denoising, noising, and why the direction matters
Patching clean activations into a corrupted run (denoising) finds sites sufficient to restore the behaviour. Patching corrupted activations into a clean run (noising) finds sites necessary to maintain it. These give different answers, and reporting one while describing the other is a common error in write-ups. A site can be sufficient in a redundant network without being necessary, and vice versa.
Path patching narrows further: instead of replacing a component's output everywhere, replace it only along the edge into one specific downstream consumer. That distinguishes "head 9.6 matters" from "head 9.6 matters because of what it sends to head 10.7".
When it breaks
It is input-specific. A heat map holds for the prompt distribution used to build it. Circuits found on templated prompts frequently do not transfer to natural text, and the templating itself can create the structure being measured.
Off-distribution activations. A patched activation is one the model would never produce in that context. The resulting behaviour can be an artefact of a strange input rather than evidence about normal operation.
Redundancy hides causes. Backup heads take over when a primary is patched out, so a genuinely important component can score near zero. This was observed directly in the IOI circuit and is the standard reason single-component ablation understates importance.
Combinatorics. Exhaustive patching over sites, positions and pairs is quadratic at best. Automated circuit discovery exists to prune the search, and the pruning heuristics are themselves assumptions about what circuits look like.
6 flashcards for this concept
Click a card to reveal the answer.