Circuit Tracing and Attribution Graphs

A sparse autoencoder tells you that a "Texas" feature fired on token 9. It does not tell you whether the Texas feature caused the model to say Austin, or whether both were caused by something else. Feature dictionaries give you a vocabulary; they do not give you a mechanism.

Circuit tracing closes that gap by building a replacement model whose internal influences are readable, then reading them (Ameisen et al., Circuit Tracing: Revealing Computational Graphs in Language Models, Transformer Circuits, 27 March 2025).

The cross-layer transcoder

A transcoder is trained to predict an MLP layer's output from the residual stream at its input, unlike an SAE which reconstructs an activation from itself. A cross-layer transcoder (CLT) generalises this: features live in \(L\) layers matching the model's, each feature reads from the residual stream at its own layer, and each may write to all subsequent MLP layers.

That last property is what makes the graph tractable. With per-layer transcoders, a feature in layer 3 influences a feature in layer 9 only through everything in between. With a CLT trained jointly across layers, early features contribute directly to later reconstructions, so feature-to-feature influence becomes a weight you can compute rather than a path you must integrate.

The local replacement model

To read influences off a specific prompt, three substitutions are made. MLPs are replaced by CLT features. Attention patterns and layer-norm denominators are frozen at the values from the real forward pass, so they become constants rather than functions of the activations. Error nodes are added, one per layer and token, holding the difference between the true MLP output and the CLT reconstruction.

The result is exact on this prompt by construction, and linear. With attention frozen, a feature's preactivation is just the sum of its incoming edges, and the edge from source \(s\) to target \(t\) is

with \(a_s\) the source's activation and \(w_{s\to t}\) the virtual weight composing the source's output vector with the target's input vector through the intervening linear maps.

The graph

Nodes are features at a layer and token position, token embeddings, error nodes, and output logits. Edges are direct linear attributions. Pruning by edge weight is essential: the authors report cutting node counts by a factor of ten while losing only 20% of the behaviour explained.

Validation is causal, not correlational. Feature activations are perturbed and the forward pass rerun from that layer, and the measured downstream change is compared with the change the graph predicted. A graph that survives that test is a hypothesis with evidence; one that has not been perturbation-tested is a picture.

When it breaks

Attention is not explained. The method freezes attention patterns, so it says nothing about how QK circuits decide where to look. On prompts where the interesting computation is the attention pattern, the authors say the graphs are "essentially useless."

The replacement model is not the model. Their largest 18-layer CLT matches the underlying model's next-token completion on 50% of a diverse prompt set. Completeness scores around 0.80 mean roughly a fifth of the computation is sitting in error nodes, unexplained by construction.

Faithfulness decays with depth. Perturbation discrepancies compound: cosine similarity to the predicted effect is around 0.8 one layer after an intervention and worse further downstream.

Human cost dominates. Graphs are large and require interpretation. On the companion study, attribution graphs gave satisfying insight on about a quarter of the prompts attempted, which is a real result and also a reminder that this is not yet an automated audit.