Induction Heads

Most claims about what an attention head "does" are informed guesses backed by attention-weight visualisation. Induction heads are different: they are one of the few attention circuits that has been reverse-engineered end to end, with the actual weight computation identified and verified, and with a documented causal link to a measurable capability, in-context learning. That combination, a concrete mechanism plus a concrete behavioural signature, makes induction heads the sharpest existing example of transformer interpretability actually explaining something.

The behaviour: complete the pattern

An induction head implements a simple algorithm: given a sequence like ... A B ... A, predict B next. If token A was followed by token B earlier in the context, and A shows up again, an induction head pushes the model to predict B again. This sounds almost too simple to matter, but it is a strict, prefix-based pattern completion running over arbitrary content, not memorised training data, and it generalises immediately to novel tokens the model has never seen paired that way before. Olsson et al., 2022, from Anthropic's interpretability team, is the paper that named and characterised the mechanism in depth.

The mechanism: two heads working across two layers

A single attention head cannot implement this algorithm alone, because the computation needs two hops: first, find where the current token last appeared, then look at what came right after that earlier occurrence. Induction heads are implemented by a circuit of two heads in different layers:

1. A previous-token head in an earlier layer writes information about "what token came immediately before me" into each position's residual stream.
2. An induction head in a later layer uses that written information: its key for position j effectively encodes "the token before j," so when the current query token A matches that key, the induction head attends to position j (the earlier occurrence of A) and copies forward whatever came after j, namely B.

This is a composition of two attention operations through the residual stream, not something visible in either head's attention pattern alone. Finding it required tracing information flow across layers, which is part of why it stayed hidden until dedicated mechanistic-interpretability tooling made the composition legible.

The evidence linking it to in-context learning

Olsson et al. found something sharper than a plausible story: across many model sizes, there is a sudden, simultaneous shift early in training where induction heads form and, at almost exactly the same point, the model's in-context learning score (how much better it gets at predicting a token when given more preceding context, sometimes visible as a bump in the loss curve) jumps. The timing correlation across model scales, combined with ablation experiments (removing induction heads measurably degrades in-context pattern-following while leaving other capabilities comparatively intact), is the strongest evidence that a specific attention circuit is causally responsible for a specific, important capability, rather than merely correlated with it. See in-context-learning for what the capability itself looks like from the outside.

Why "copy the pattern" scales into "few-shot learning"

The leap from "complete A B ... A -> B" to "learn a new task from a few examples in the prompt" is less of a leap than it sounds. A few-shot prompt is, structurally, a sequence of repeated patterns: input1 -> output1, input2 -> output2, input3 ->. An induction mechanism that generalises beyond exact token matches, to matching on more abstract features (this is a translation example, this is a sentiment-labelling example), and copies the completion pattern rather than the literal tokens, starts to look a great deal like few-shot in-context learning. Olsson et al. present this generalisation, from literal token copying to abstract pattern copying, as the mechanistic seed that later composes with other circuitry into fuller in-context learning behaviour, not the entire explanation of it.

When it falls down

• It explains a mechanism, not the whole capability. Induction heads are convincingly linked to simple pattern-completion in-context learning. They are not a complete account of everything called "in-context learning," including complex few-shot reasoning, which likely involves many composed circuits beyond this one.
• The clean two-layer story is a simplification. Real models often have induction-like behaviour distributed across more than two heads, with partial or redundant implementations, and not every model exhibits a single crisp induction circuit that ablation cleanly removes.
• Formation timing is not fully understood. The correlation between induction head formation and the in-context learning phase transition is well documented, but why training dynamics produce this particular circuit at this particular point, rather than some other pattern-matching solution, remains an open question.
• Interpretability findings are architecture and scale dependent. The clearest induction head evidence comes from specific model families studied by specific interpretability teams; the mechanism is believed to generalise broadly, but claims about its exact form in any particular production model are inference, not direct verification, unless that model has itself been probed.

Further reading

• Olsson et al., 2022, In-context Learning and Induction Heads, transformer-circuits.pub - the paper that named, characterised, and causally tested induction heads.
• Elhage et al., 2021, A Mathematical Framework for Transformer Circuits, transformer-circuits.pub - the circuit-composition framework (QK and OV circuits) induction heads are described within.