From Features to Circuits: What Attribution Graphs Explain, and the Fraction They Do Not

Give Claude 3.5 Haiku the prompt "Fact: the capital of the state containing Dallas is" and it answers Austin. The interesting question is not whether it is right. It is whether the model retrieved a memorised association or computed one, and until March 2025 there was no method that could answer that in a production-scale model.

The answer turned out to be: it computed one. Dallas features activate Texas features, Texas features combine with capital features to activate say-Austin features. And the evidence is not that the graph looks like that. The evidence is that if you substitute California features for the Texas features the model says Sacramento, substitute Georgia and it says Atlanta, substitute British Columbia and it says Victoria, substitute the Byzantine Empire and it says Constantinople (Lindsey et al., On the Biology of a Large Language Model, Transformer Circuits, 27 March 2025).

Why this matters: Interpretability spent five years learning to name what is inside a model. Circuit tracing is the first scalable attempt to describe what a model does, causally, on a specific input. That distinction decides whether interpretability can ever support an audit, and the honest accounting of how much it currently explains is more useful than the highlight reel.

TL;DR

• A cross-layer transcoder replaces a model's MLPs with features that read at one layer and write to all later layers, which turns feature-to-feature influence from a path you must integrate into a weight you can compute.
• Freezing attention patterns and layer-norm denominators makes the replacement model linear on that prompt, so an edge is exactly \(A_{s\to t} = a_s w_{s\to t}\) and a node's preactivation is the sum of its incoming edges.
• The Dallas → Texas → Austin trace is validated by substitution, not inspection. Five different geographic swaps produce five correct different capitals.
• On poetry, the model selects rhyme candidates before writing the line. Injecting a planned word made the model end the line with it in 70% of cases.
• The limits are stated by the authors: the largest 18-layer CLT matches the underlying model's next-token completion on 50% of a diverse prompt set, completeness scores sit around 0.80, and attribution graphs gave satisfying insight on about a quarter of attempted prompts.
• Attention is not explained at all. The method freezes QK patterns to gain linearity, so prompts whose mechanism is the attention pattern produce graphs the authors call "essentially useless."
• Meanwhile, on tasks with a known target concept, SAEs lose to much cheaper baselines. AxBench found prompting beats every method at steering and difference-in-means beats SAEs at concept detection (Wu et al., arXiv:2501.17148).

At a Glance

flowchart LR
    M["Real forward pass<br/>on one prompt"] --> F["Freeze attention<br/>and LayerNorm denominators"]
    M --> C["Cross-layer transcoder<br/>replaces MLPs"]
    F --> R["Local replacement model<br/>linear, exact on this prompt"]
    C --> R
    R --> E["Error nodes<br/>unreconstructed MLP output"]
    R --> G["Attribution graph<br/>features, embeddings, logits"]
    G --> P["Perturbation test<br/>predicted vs measured effect"]
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class M blue
    class F,C,R purple
    class G,P teal
    class E amber

The amber box is the honest one. Everything the transcoder failed to reconstruct is parked in error nodes, and error nodes are unexplained by construction.

Before Circuits, Dictionaries

The field arrived here by two routes that only recently joined.

The first was circuit analysis by hand. A mathematical framework for reverse-engineering attention-only transformers appeared in December 2021, introducing QK and OV circuits as separable objects (Elhage et al., Transformer Circuits). Causal methods followed: Meng et al. localised factual recall to mid-layer feed-forward modules at the subject token and then edited it with ROME (NeurIPS 2022, arXiv:2202.05262); Wang et al. isolated a 26-head, 7-group circuit for indirect object identification in GPT-2 small (arXiv:2211.00593). These worked, and they did not scale. Each was months of human effort on one behaviour in a small model.

The second route was dictionary learning, motivated by superposition: a model with more concepts than neurons represents them as overlapping directions, so no neuron means one thing (Elhage et al., Toy Models of Superposition, 2022, arXiv:2209.10652). Sparse autoencoders were proposed as the decomposition, demonstrated on a one-layer model in October 2023, then scaled: up to 34 million features on Claude 3 Sonnet's middle-layer residual stream (Templeton et al., Scaling Monosemanticity, May 2024), 16 million latents trained on GPT-4 activations over 40 billion tokens (Gao et al., arXiv:2406.04093), and JumpReLU SAEs released for every layer and sublayer of Gemma 2 2B and 9B (Lieberum et al., Gemma Scope, arXiv:2408.05147).

Dictionaries scaled and circuits did not. What dictionaries could not do was say which feature caused which.

timeline
    title Two lines of interpretability converging
    2021 : A Mathematical Framework for Transformer Circuits separates QK and OV
    2022 : ROME localises factual recall with causal tracing
         : Interpretability in the Wild finds a 26-head IOI circuit by hand
         : Toy Models of Superposition explains why neurons are polysemantic
    2023 : Towards Monosemanticity decomposes a one-layer model with an SAE
    2024 : Scaling Monosemanticity reaches 34M features on Claude 3 Sonnet
         : OpenAI trains a 16M-latent SAE on GPT-4; Gemma Scope releases SAEs for all layers
    2025 : January - AxBench finds SAEs uncompetitive against prompting and difference-in-means
         : February - sparse probing study finds SAEs do not beat logistic regression
         : March - Circuit Tracing and On the Biology of a Large Language Model
         : May - Anthropic open-sources circuit-tracer with a Neuronpedia frontend

Read the 2025 entries together. In the same quarter the field published its most convincing mechanistic result and its most damaging negative result about the same underlying tool.

[IMAGE: Two-panel figure. Left: a neuron activating on Chinese text, cat pictures, and Python decorators, labelled "polysemantic". Right: three separate feature directions each activating on one of those, labelled "monosemantic", drawn as arrows in a 2D projection with the neuron basis shown as faint grey axes. Caption: "Superposition is why the neuron basis is the wrong coordinate system."]

How a Replacement Model Is Built

Three ideas compose, and each is doing a specific job.

A transcoder predicts output, not input

An SAE reconstructs an activation from itself. It answers "what is present here". A transcoder is trained to predict an MLP layer's output from the residual stream at its input. That is a different object: it stands in for the computation rather than describing the state, which means you can remove the MLP and put the transcoder in its place and still have a working forward pass.

Cross-layer, so influence is a weight

A cross-layer transcoder organises features into \(L\) layers matching the model's. Each feature reads from the residual stream at its own layer, and each may contribute output to all subsequent MLP layers. Because the layers are trained jointly rather than independently, an early feature contributes directly to a later reconstruction.

This is the structural move that makes graphs tractable. With per-layer transcoders, the influence of a layer-3 feature on a layer-9 feature runs through everything between them and has to be integrated. With a CLT it is a single virtual weight.

Freeze attention to buy linearity

To analyse one prompt, build a local replacement model: substitute CLT features for MLPs, freeze the attention patterns and layer-norm denominators at the values the real forward pass produced, and add error nodes holding the difference between each true MLP output and its reconstruction.

The frozen quantities were functions of the activations; now they are constants. What remains is, in the authors' phrasing, "a very large fully connected neural network, spanning across tokens", and it reproduces the original output on this prompt exactly, because the error nodes absorb whatever the transcoder missed.

Linearity is the payoff. A feature's preactivation is simply the sum of its incoming edges, and the edge from source \(s\) to target \(t\) is

where \(a_s\) is the source's activation on this prompt and \(w_{s \to t}\) is the virtual weight composing the source's output vector with the target's input vector through the intervening linear maps. No approximation, no gradient estimate, no path integral. The decomposition is complete: every unit of a target's preactivation is attributed to some incoming edge or to an error node.

Prune, then test causally

Raw graphs are unreadable. Pruning by edge weight is reported to cut node counts by a factor of ten while reducing the behaviour explained by only 20%, which is the working assumption of the whole method: influence is concentrated in a small fraction of edges.

Then the graph is tested rather than admired. Constrained patching: modify a feature's activation, run the forward pass from that layer, and compare the measured downstream change against what the graph predicted. A graph that survives is a causal hypothesis with evidence. One that has not been perturbed is a picture of a linear algebra fact.

Seeing It in Motion

The Dallas graph, reduced to its load-bearing nodes:

flowchart TB
    D["Token: Dallas"] --> DF["Dallas features"]
    S["Token: state"] --> SF["State features"]
    K["Token: capital"] --> KF["Capital features"]
    DF --> TX["Texas features"]
    SF --> TX
    KF --> SAY["Say-a-capital features"]
    TX --> AUS["Say-Austin features"]
    SAY --> AUS
    AUS --> L["Logit: Austin"]
    ERR["Error nodes"] -. unexplained .-> L
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class D,S,K blue
    class DF,SF,KF,TX,SAY,AUS purple
    class L teal
    class ERR amber

The Texas node is the claim. It is not in the prompt and not in the output, so if the model were doing pure associative retrieval it would not need to exist.

The validation loop is what turns that from a story into a result:

sequenceDiagram
    participant A as Analyst
    participant G as Attribution graph
    participant M as Model
    A->>G: Identify candidate intermediate node
    G-->>A: Predicted effect on logits
    A->>M: Suppress or replace that feature
    M-->>A: Measured output change
    A->>A: Compare predicted vs measured
    Note over A,M: Discrepancy compounds with depth;<br/>cosine similarity around 0.8 one layer downstream

And the state of a single analysis, which is where the human cost hides:

stateDiagram-v2
    [*] --> BuildGraph
    BuildGraph --> Prune
    Prune --> Interpret
    Interpret --> Perturb
    Perturb --> Confirmed: prediction matches
    Perturb --> Interpret: prediction fails, revise
    Interpret --> Abandoned: no legible structure
    Confirmed --> [*]
    Abandoned --> [*]

Roughly three of every four attempted prompts end in Abandoned.

By the Numbers

Sources: Circuit Tracing methods, On the Biology of a Large Language Model, Scaling Monosemanticity, Gao et al., Wang et al., Arditi et al., Wu et al., Anthropic open-source release. The completeness, faithfulness and insight-rate figures are the authors' own self-assessments and have not been independently replicated at the time of writing.

A Concrete Example

Walk the Dallas prompt through the machinery, with the arithmetic made explicit. Activation values below are illustrative stand-ins chosen to show the shape of the computation; the qualitative structure and every intervention outcome are from the published study.

Step 1. Run the model. Prompt: Fact: the capital of the state containing Dallas is. Output: Austin. Cache all activations, attention patterns and normalisation denominators.

Step 2. Build the local replacement. Substitute CLT features for every MLP. Freeze the cached attention patterns. For each layer and token, compute the reconstruction residual and store it as an error node. The replacement now reproduces Austin exactly.

Step 3. Read the active features at the final token. Suppose the say-Austin feature cluster has a preactivation of \(2.30\), decomposed by incoming edge:

Two things to notice. The Texas path dominates, and the direct Dallas-to-Austin edge exists but is small, which is the quantitative version of "it is not pure retrieval". The 13% in error nodes is the part of the answer this method does not explain, and it is typical.

Step 4. Test the Texas node by substitution. Replace the Texas features with the features for another region and rerun from that layer. Published outcomes:

Five substitutions, five correct capitals for the substituted entity, none of which appear anywhere in the prompt. The authors note that non-US entities required larger injection magnitudes, which is itself informative: the circuit is better fitted to US geography, presumably reflecting the training distribution.

Step 5. Check that the effect is specific. Suppressing the Texas features without substituting anything degrades the Austin logit rather than redirecting it, which is the pattern a genuine intermediate should show. A node that were merely correlated would be removable without changing the output at all.

Step 6. Generalise the method, not the result. The same procedure applied to poetry found the model selecting rhyme candidates before writing the line. Injecting a planned word made the model end the line with it in 70% of cases, and suppressing the "rabbit" features changed the line's ending to match the alternative planned word, "habit". Planning ahead inside a single forward pass is a claim about architecture that would be very hard to establish any other way.

[IMAGE: Sankey-style diagram of the say-Austin preactivation, with four inbound bands (Texas 48%, say-a-capital 27%, direct Dallas 12%, error 13%) whose widths match the table, converging on a single node. Caption: "Every unit of the preactivation is attributed. The amber band is the part the method cannot explain."]

Where It Breaks

Attention is outside the theory

The linearity that makes the graph readable was bought by freezing attention. So the method describes what flows through attention and says nothing about how the QK circuit decided where to look. The authors are direct about the consequence: there are cases where the attribution graphs are "essentially useless", specifically those where the interesting computation is the pattern itself. Induction, retrieval from a long context, and most of the behaviours that make long-context models interesting fall in this gap.

The replacement model is not the model

An 18-layer CLT matching the underlying model's next-token completion on 50% of a diverse prompt set is a real result and also a hard ceiling. Half the time, the object being explained gives a different answer from the object you care about. Completeness around 0.80 means roughly a fifth of the computation is sitting in error nodes, and error nodes are not a residual you can shrink by trying harder; they are where the method's assumption about MLP decomposability fails.

Faithfulness decays with depth

Perturbation discrepancies compound. Cosine similarity between predicted and measured effect is around 0.8 one layer after an intervention and worse further downstream. For a shallow circuit that is tolerable. For a claim spanning twenty layers, the error budget is spent long before the output.

Interpretation does not automate

Attribution graphs gave satisfying insight on about a quarter of attempted prompts. The other three quarters are not failures of the model; they are graphs with no legible structure, or with structure that nobody has yet worked out how to read. Auditing a model means covering behaviours you did not choose, and a method with a 25% hit rate on chosen prompts is not yet that.

The dictionaries underneath lose to cheap baselines on known concepts

This is the sharpest tension in the field and it deserves stating without softening. AxBench evaluated steering and concept detection on Gemma-2-2B and Gemma-2-9B across prompting, finetuning, SAEs, linear probes, difference-in-means and representation finetuning. Result: prompting outperforms every method at steering, finetuning second, and SAEs are uncompetitive on both tasks; for concept detection, difference-in-means wins (Wu et al., arXiv:2501.17148). A separate sparse-probing study reached a compatible conclusion, finding SAEs failing to consistently beat logistic-regression probes, including in the data-scarce and label-noise regimes where they were expected to help (Kantamneni et al., arXiv:2502.16681).

The reconciliation is not that one side is wrong. It is that these measure different jobs. If you already know the concept, a probe or a steering vector trained on it will beat a general dictionary, because the dictionary spent its capacity on the other 33,999,999 features. Dictionaries and circuits are discovery instruments. Probes and steering vectors are acting instruments. Benchmarking a discovery instrument on an acting task will always flatter the baseline.

Feature splitting makes "a feature" ambiguous

Train a larger dictionary on the same activations and a single feature resolves into several finer ones. There is no canonical granularity, so "the Texas feature" is really "the Texas cluster at this dictionary size", and circuits built at one width do not necessarily correspond node-for-node to circuits built at another.

Alternative Designs

The practical stack in 2026 uses several of these in sequence: dictionaries or transcoders to propose a mechanism, patching to test it, probes to monitor it in production, and steering to intervene. Treating any one of them as the method is what produces both the overclaiming and the backlash.

How It Is Used in Practice

Anthropic open-sourced the tooling on 29 May 2025, with the work led by participants in their Fellows program in collaboration with Decode Research. The circuit-tracer library generates attribution graphs on supported open models, initially Gemma-2-2B and Llama-3.2-1B, and Neuronpedia hosts an interactive frontend for exploring, annotating and sharing them, including modifying feature values and observing the output change (Anthropic, Open-sourcing circuit-tracing tools). That release matters more than a typical code drop, because before it the method's evidence base was entirely internal to one lab.

The safety-relevant applications in the biology paper are the ones worth watching. Attribution graphs were used to examine refusals, jailbreaks, hallucination and entity recognition, chain-of-thought faithfulness, and a deliberately misaligned model variant carrying a hidden goal. The last of these is the closest thing to an audit that currently exists: a model organism was trained with a known undesired objective, and the question was whether interpretability could recover it without being told.

What has not happened is the obvious next step. There is no standard interpretability report attached to a model release, no agreed coverage metric, and no answer to "what fraction of this model's behaviour has been mechanistically explained" that anyone would defend. The 25% insight rate and the 0.80 completeness score are the field's own numbers and they are, for now, the honest headline.

[IMAGE: Screenshot-style mockup of an attribution graph browser: a layer-by-token grid with feature nodes sized by activation, edges weighted by attribution, a right-hand panel showing top activating examples for the selected feature, and an intervention slider. Caption: "The interface is the method. A graph nobody can navigate explains nothing."]

Insights Worth Remembering

1. Substitution is the evidence, not the graph. The Dallas result is convincing because five different entity swaps produced five different correct capitals. A graph without a perturbation test is a description of a linear decomposition, which is true by construction and therefore says nothing.

2. A transcoder is a different object from an SAE, and the difference is the whole method. An SAE describes a state. A transcoder predicts a layer's output from its input, so it can replace the layer. Only the replacement lets you compute influence rather than correlation.

3. Linearity was purchased, and the price was attention. Freezing QK patterns is what makes edges exact. It is also why the method is silent on retrieval, induction and long-context behaviour, which is a large fraction of what makes modern models useful.

4. Error nodes are a measurement, not a nuisance. A completeness score of 0.80 says a fifth of the computation resists this decomposition. Reporting the graph without the error mass is the interpretability equivalent of reporting accuracy without a confidence interval.

5. Discovery instruments and acting instruments should not be benchmarked against each other. SAEs lose to difference-in-means on known concepts because the dictionary spent its capacity elsewhere. That is a design consequence, not a refutation, and reading it as a refutation has cost the field more clarity than the negative results themselves.

6. Models plan inside a forward pass. The poetry result, with a 70% intervention success rate, is a claim about architecture that no behavioural evaluation could establish: the rhyme word is selected before the line that leads to it is written.

7. Circuits found at one dictionary width may not exist at another. Feature splitting means granularity is a hyperparameter, so a circuit is a statement about a model and a dictionary. Papers that omit the dictionary are underspecified.

8. A 25% insight rate is a real result and an inadequate audit. Both halves matter. It is enormous progress over hand-built circuits that took months each, and it is nowhere near the coverage that a safety case would require.

Open Questions

Can attention circuits be brought inside the framework? Measured: freezing attention yields exact edges. Not established: whether any decomposition of the QK circuit preserves that exactness. This is the single largest gap, and closing it would roughly double the behaviours the method can address.

Does the 25% insight rate improve with tooling or is it a property of the models? It is currently unknown whether the three-quarters that fail are illegible because graphs are hard to read, because the CLT reconstruction is poor on those prompts, or because the underlying computation genuinely has no sparse mechanistic description. These have very different implications, and no published work separates them.

Is there a canonical feature granularity? Feature splitting suggests not. If there is no privileged width, then "the model's circuit" is not well defined, and interpretability results need to report dictionary size the way ML results report seed and dataset.

Do circuits transfer across models? The biology paper studied Claude 3.5 Haiku; the open-source tooling targets Gemma-2-2B and Llama-3.2-1B. Whether the Dallas-style two-step structure is a universal solution to that task shape or an artefact of one training run is untested, and it decides whether interpretability findings accumulate or expire with each model release.

What would a sufficient interpretability audit even look like? No coverage metric exists. Completeness of 0.80 on selected prompts is not a statement about a model's behaviour in deployment, and until there is an agreed target, "we interpreted the model" remains a claim without a threshold.

Sources and Further Reading

The circuit tracing papers

1. Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Chen, B., Citro, C., et al. (2025). "Circuit Tracing: Revealing Computational Graphs in Language Models." Transformer Circuits Thread, 27 March 2025. transformer-circuits.pub/2025/attribution-graphs/methods.html
2. Lindsey, J., Batson, J., et al. (2025). "On the Biology of a Large Language Model." Transformer Circuits Thread, 27 March 2025. transformer-circuits.pub/2025/attribution-graphs/biology.html
3. Anthropic (2025). "Open-sourcing circuit-tracing tools." 29 May 2025. anthropic.com/research/open-source-circuit-tracing

Dictionary learning and superposition

4. Elhage, N., Hume, T., Olsson, C., et al. (2022). "Toy Models of Superposition." Transformer Circuits Thread. arXiv:2209.10652
5. Bricken, T., Templeton, A., Batson, J., et al. (2023). "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning." Transformer Circuits Thread, October 2023. transformer-circuits.pub/2023/monosemantic-features
6. Templeton, A., Conerly, T., Marcus, J., et al. (2024). "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet." Transformer Circuits Thread, May 2024. transformer-circuits.pub/2024/scaling-monosemanticity
7. Gao, L., Dupré la Tour, T., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., & Wu, J. (2024). "Scaling and evaluating sparse autoencoders." arXiv:2406.04093
8. Lieberum, T., Rajamanoharan, S., Conmy, A., et al. (2024). "Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2." arXiv:2408.05147

Causal methods and hand-built circuits

9. Elhage, N., Nanda, N., Olsson, C., et al. (2021). "A Mathematical Framework for Transformer Circuits." Transformer Circuits Thread, 22 December 2021. transformer-circuits.pub/2021/framework/index.html
10. Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). "Locating and Editing Factual Associations in GPT." NeurIPS 2022. arXiv:2202.05262
11. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., & Steinhardt, J. (2022). "Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small." arXiv:2211.00593
12. Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., & Nanda, N. (2024). "Refusal in Language Models Is Mediated by a Single Direction." arXiv:2406.11717

The negative results

13. Wu, Z., Arora, A., Geiger, A., Wang, Z., Huang, J., Jurafsky, D., Manning, C. D., & Potts, C. (2025). "AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders." arXiv:2501.17148
14. Kantamneni, S., Engels, J., Rajamanoharan, S., Tegmark, M., & Nanda, N. (2025). "Are Sparse Autoencoders Useful? A Case Study in Sparse Probing." arXiv:2502.16681

Steering and representation control

15. Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., & MacDiarmid, M. (2023). "Steering Language Models With Activation Engineering." arXiv:2308.10248
16. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., et al. (2023). "Representation Engineering: A Top-Down Approach to AI Transparency." arXiv:2310.01405

Related concepts in this library

17. Circuit Tracing and Attribution Graphs
18. Activation Patching and Causal Tracing
19. Activation Steering and Representation Engineering
20. Sparse Autoencoders for Interpretability

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.