Dissection · Olsson et al., 2022

Anatomy of Induction Heads

In-context Learning and Induction Heads, dissected: a two-layer attention-only Transformer in 3D, every matrix a real one from a model trained in the repository, with seven pre-registered hypotheses tested on 14 training runs.

PaperOlsson et al., Transformer Circuits Thread, 2022

Model45,376 parameters, live

Reading24 min · 30 steps

Published2026-10-20

01Overview

How does a language model use something it has seen once?

Show a model the sequence k q … k and it becomes likelier to continue with q. Olsson et al. (2022) [1] describe the circuit behind that habit, the induction head, as a "simple algorithm to complete token sequences like [A][B] … [A] → [B]" (their abstract), and argue for the hypothesis that these heads might be the mechanism for the majority of all in-context learning in large Transformers. It is the founding result of mechanistic interpretability on in-context learning; the library has a concept page on induction heads.

This page builds the smallest model in which that circuit forms: two attention layers, no feed-forward layers, parameters, trained on this page's own data and running in your browser on whatever you type. The 3D stage shows its every matrix as a block of cells, with layer 1 at the lower left, layer 2 up and to the right, and the residual stream climbing between them. Scroll to move through the computation, in order, then through training, then through the evidence, then through three pre-registered experiments on its claims.

02Overview

What is running here, and how does it differ from the paper's models?

The paper's small models are one- to six-layer Transformers, 768 wide with 12 heads per layer and, in the attention-only series, no MLPs (Model Details); they were trained for 10,000 steps, about 10 billion tokens. The miniature keeps the architecture (attention-only, LayerNorm before each layer, separate embedding and unembedding) and shrinks the rest, as the table shows.

You typed tokens, a start token ^ plus: . The start token is there because the paper's own test also puts one in front (Methods). The model reads at most 64 tokens.

03Overview

What did the model read while it trained?

Web text has local statistics (which characters follow which) and repetition (a name used twice). The miniature's data has both, made to order. A fixed Markov chain, with eight likely successors for every character, is the background. Now and then a span of 4 to 10 characters starts; it is either new, made of uniformly random characters, or a verbatim repeat of an earlier new span in the same sequence. Looking back pays only inside repeats, and only after their first character.

The random spans are a design choice with a reason. A first version copied spans from the chain's own text, and the model learned in-context gains without any induction head (a skip-trigram shortcut: chain-consistent continuations can be guessed from the current character alone; the research desk's lab note has the numbers). Random spans are also the distribution of the paper's own test, "a repeated sequence of completely random tokens" (Key Concepts). Bietti et al. [2] use a similar synthetic setup, "tokens … generated from either global or context-specific bigram distributions" (abstract), and see the same order of learning: fast learning of global bigrams, then the slower development of an induction head.

04Overview

How do you measure learning that changes no weight?

Kaplan et al. [3] look at the next-token loss at each position in the context; the paper adopts this as its formalism and defines a one-number summary: the loss of the 500th token minus the average loss of the 50th. The paper averages over 10,000 random examples of 512 tokens; the page uses 1,024 held-out sequences of 64. The positions were chosen "somewhat arbitrarily", and the paper contrasts this with the few-shot conception of Brown et al. [4], which it sets aside.

The page uses the same idea at its own scale: the mean loss at tokens 8 to 15, minus the mean loss at tokens 56 to 63, of a 64-token context.

ICL = \overline{ℓ}_{8 - 15} - \overline{ℓ}_{56 - 63}, ℓ_{i} = - lo g p_{θ} (t_{i} ∣ t_{< i})

It is positive when later tokens are easier. The paper's two statements of the score disagree about the order of the subtraction (see the notes on the paper below); the page uses the order of its prose, "the 50th token loss minus the 500th token loss".

Before its phase change the paper's models have less than 0.15 nats of in-context learning, afterwards roughly 0.4. The published model on this page ends at nats, larger because this data is repeat-heavy.

05Layer 1

Where does a character become a vector?

The model has a table with one learned 64-dimensional row for each of its 65 tokens, the start token and the 64 characters. Reading a character copies its row into the residual stream, the running record each layer adds to.

x_{0} = E [t], E \in R^{65 \times 64}

The highlighted rows of E on the stage are the characters of your text; each teal line is one lookup. The paper's small models have the same table with 2¹⁶ rows and 768 columns (Model Details). Nothing about the position of a character is in x₀.

06Layer 1

Where does the model learn the order of the characters?

It learns a vector for each position (see also positional encodings), and adds it to the queries and keys of every layer, never to the residual stream. The paper describes its choice only as "a variant on the standard positional embeddings (similar to Press et al.)" (Model Details) and states the consequence: its models "do not add positional information into the residual stream" (Argument 5). The page reads this as the positional-infused attention of Press et al. [5], which adds position vectors to queries and keys at every layer.

\tilde{x} = LN (x) + P, Q = \tilde{x} W^{Q}, K = \tilde{x} W^{K}, V = LN (x) W^{V}

The block P on the stage is the first of its 64 rows. The consequence matters for this circuit: because the residual stream carries no position, the only way layer 2 can find out what came before a token is for an earlier head to write it there.

07Layer 1

Why normalise before attention?

Each row of the residual stream is centred and rescaled by LayerNorm before it is used, so that attention sees the direction of a token's vector and not its length. The page's LayerNorm has a learned gain and no bias.

LN (x) = \frac{x - μ ( x )}{σ ^{2} ( x ) + ϵ} ⊙ g

The paper notes that in an attention-only model the logits are a sum of per-head terms "up to a rescaling due to LayerNorm", which is why ablating one head at a time is a principled measurement there.

08Layer 1

What does each character ask, advertise and carry?

Three learned matrices turn each row into a query (what it is looking for), a key (what it offers) and a value (what it hands over if chosen). Each is split into four heads of 16 dimensions; the paper's heads are 64 wide, twelve of them (Model Details). The stage computes one cell first, a row of the input times a column of W^Q, then fills the rest.

Queries and keys read the position-augmented input and values read the normalised residual alone. A head's job is therefore split in two: where to look (queries and keys) and what to take (values), and the paper stresses that the two are separable circuits.

09Layer 1

How much does each token look at each earlier token?

A_{h} = softmax (\frac{Q _{h} K _{h}^{⊤}}{d _{h}} + M), M_{ij} = {0 - \infty j \leq i j > i

A score is the dot product of one query with one key, scaled by the square root of the head width as in the original Transformer [6]. The causal mask sets every later position to −∞, so a token can only look back; the softmax then turns each row into weights that sum to 1 (softmax; the attention mechanism in general). The stage shows four heads, each as scores then weights.

10Layer 1

What is a head that only ever looks one token back good for?

The paper's circuit has two parts: "the first head is a 'previous token head' which copies information from the previous token into the next token, while the second head (the actual 'induction head') uses that information to find tokens preceded by the present token" (Introduction). In layer 1 of this model one or two heads draw a band just below the diagonal of their weight matrices: attention on position i−1.

The published model's layer 1 heads have previous-token scores of on training-like text, measured as in the paper: the mean attention from token i to token i−1. None is a clean previous-token head, and the paper says of its own heads that "in practice, induction heads don't exhibit these properties perfectly".

11Layer 1

How do four heads write into one stream?

This is multi-head attention in an attention-only model.

z_{h} = A_{h} V_{h}, out = Concat (z_{1}, \dots, z_{4}) W^{O}, x_{1} = x_{0} + out

Each head takes a weighted average of its values, with the weights from the softmax; the four results are laid side by side, multiplied by W^O and added to the residual stream. In this model that is the only thing a layer does, so everything layer 2 will know about layer 1 is in x₁.

A head whose weights sit on position i−1 hands token i a value that is a function of token i−1. After the addition, the row for token i in the residual stream carries both its own identity and a copy of its predecessor's.

12Layer 2

How does layer 2 find out what came before each token?

By reading it. Layer 2 computes its keys from x₁, which now holds each token's predecessor. A query carrying the current token A therefore scores highest against the key of a token whose predecessor was also A: the token that followed A last time. The paper calls this K-composition and gives the reason layers cannot be skipped in a footnote: a single head's score for an attended token cannot depend on the token before it.

The same footnote is why the paper finds that induction heads "don't occur in 1 layer models". The figure cuts the route: layer 2's keys are recomputed from the token embeddings alone, with everything else unchanged.

The orange line on the stage is that route: from what layer 1 wrote, to the keys of layer 2.

13Layer 2

What do layer 2's queries and keys contain?

The same three products, now on x₁. A layer 2 query, at token i, still carries the identity of token i. A layer 2 key, at position j, carries both the token at j and (written by layer 1) the token at j−1. Their dot product is large when the query's token matches the key's predecessor.

The stage fills queries, keys and values for layer 2 in that order. The weights, W^Q, W^K and W^V, are different from layer 1's and have 4,096 numbers each (64 by 64).

14Layer 2

Where does a layer 2 head look?

The paper defines an induction head by its behaviour on repeated random tokens (Key Concepts). It has two parts: prefix matching, attending back to previous tokens that were followed by the current and/or recent tokens, and copying, raising the logit of the attended-to token.

In the repeated half of a text, the pattern is a stripe parallel to the diagonal, offset by the length of the repeat. The outlined cells mark the induction sources. The paper's test sequence is 25 random tokens, repeated four times behind a start token (Methods); the research desk uses the same test with 15 tokens, to fit a 64-token context.

15Layer 2

What does an induction head write?

The attended token's own value, pushed through W^O into a direction that raises that token's logit. Following a head's output straight to the logits, the effect of attending to token a on the logit of token b is a 64 by 64 table. A copying head is one whose table is largest on the diagonal.

effect_{h} (a \to b) = [LN (E [a]) W_{h}^{V} W_{h}^{O} U]_{b}

This is the page's version of the paper's copying evaluator, which scales a ratio into [−1, 1]. The paper describes its ratio, the amount a head raises the logit of the attended token against the amount it raises all tokens in the sample, as ranging from 0 (only raises other tokens) to 0.5 (only raises the present token), which cannot be right for a share of a total. The page's score, used in the research desk, is the contrast between the attended token's boost and the others'.

16Output

How does the stream become a prediction?

A final LayerNorm, then (the unembedding) a learned 64 by 65 matrix U that turns each residual row into a logit for every token, then a softmax. U is a separate matrix, not tied to E.

p_{θ} (\cdot ∣ t_{\leq i}) = softmax (LN (x_{2}) U)_{i}, ℓ_{i} = - lo g p_{θ} (t_{i + 1} ∣ t_{\leq i})

On your text, the model's choice after the last character is . Each row of the stage's probability block is one prediction, scored against the character that actually came next; the loss is how surprised the model was.

17Output

Two heads, one algorithm: how does [A][B] … [A] become [B]?

Take your text and follow one repeat. (1) A layer 1 head, by attending to position i−1, copies each token's predecessor into its residual row. (2) A layer 2 query at the second A compares itself with every key; the key at the token after the first A contains A as its predecessor, so it matches. (3) The head's value at that position is B's identity. (4) W^O and U turn it into a raised logit for B.

The paper, building on its predecessor [8], reads the same circuit from the weights: a "copying" matrix with positive eigenvalues for step 4, and K-composition with a previous-token head for step 2. Redwood Research replicated it in two-layer attention-only models: replacing the previous-token head's attention pattern with an exact previous-token pattern "recovered 99% of the loss difference", and the induction head's with a simple approximation, about 65%.

The induction pattern is a small instance of a general move the paper calls a "fuzzy" or "nearest neighbor" version of pattern completion. Everything above is exact, because the page's model is small enough to compute exactly; what the paper adds for large models is argued below.

18Training

How was the miniature trained?

On next-token prediction, which is the only signal the model gets.

L (θ) = - \frac{1}{T - 1} i = 1 \sum T - 1 lo g p_{θ} (t_{i + 1} ∣ t_{\leq i})

The batch was 64 sequences of 64 tokens, the optimiser Adam with β = (0.9, 0.95), a peak learning rate of 3·10⁻³ reached by linear warm-up over 150 steps, no weight decay, no dropout, and 3,000 steps in all, which took about minutes on one CPU core. The paper's small models warmed up over the first 1.5 billion tokens, which it notes falls within the range of the phase change, and had their weight decay reduced at 4,750 steps, after it. Here the warm-up ends at step 150, long before the phase change below.

19Training

What do the heads look like before, during and after training?

The stage replays training: the attention blocks of both layers, on your text, are rewritten from eighteen snapshots of the published run, from step 0 to step 3,000. At the start every head is a smear. For a long plateau nothing much changes. Then, from about step 1,100 for this run, the layer 2 stripe appears, and the layer 1 band below the diagonal strengthens with it.

The order the figure shows, fast learning of the local statistics followed by a slower, later development of the induction mechanism, is what Bietti et al. describe in their synthetic setting global bigrams fast, the induction head slowly. Reddy [9] traces the abruptness to the sequential learning of three nested logits: attending to the right place in layer 1, in layer 2, and then trusting the result.

20Training

What moves together during the phase change?

The paper's first argument is a coincidence in time. During one window, which it puts at roughly 2.5 to 5 billion tokens, four things happen at once: the in-context learning score rises, induction heads form, the training loss shows a visible bump, and the model's trajectory in the space of per-token losses turns (its summary of the window). The bump is "the only place in training where the loss is not convex".

On the published run, The research desk below does this for five seeds and states in advance what "together" means.

21The evidence

What does the paper claim, and how strongly?

The paper's evidence comes from 34 models and more than 50,000 head ablations, organised as six arguments, each with its own strength rating for each model class. Of the first three, for small attention-only models, the paper says it believes it has strong evidence. The abstract's wording: strong, causal evidence for small attention-only models and correlational evidence for larger ones [1].

Argument 1 is the coincidence above. Argument 2 changes the architecture so the phase change should move (smeared keys). Argument 3 removes the heads and watches the in-context learning score fall. The research desk re-runs these three on a model small enough to run repeatedly.

22The evidence

What happens to the prediction when a head is removed?

Ablation is the cousin of activation patching: remove a component, measure what breaks.

The paper's ablation, in its own words: run the model once and save the attention patterns; run it again with the ablated head's result vector replaced by zeros and all attention patterns forced to the recorded ones. Pattern-preserving ablation removes the head's contribution to what later heads read but leaves intact its contribution to where they look. The result for small attention-only models: "almost all the in-context learning … appears to come from these induction heads".

23The evidence

Does the same circuit explain more than copying random tokens?

This is Argument 4, and it is the one the miniature cannot test. The paper shows, in a 40-layer, 13-billion-parameter model, heads that satisfy the narrow definition and also complete translations and a templated pattern; the pattern-matching head allocates about 65% of its attention from the colons to the correct positions. The claim is that the same [A*][B*] … [A] → [B] mechanism works over similar tokens as well as identical ones.

Argument 6 extends the story from small to large models: the models all have the same sharp increase in in-context learning, with the same rough amounts before and after the transition. The full-scale series runs from 4 layers and 13M non-embedding parameters to 40 layers and 13B, with width d_model = 128 × layers. A layer with a four-times-wide MLP has 12d² weights, and 12·4·512² ≈ 12.6M and 12·40·5120² ≈ 12.6B reproduce the paper's two ends of that column.

Two later papers bear on generality. Crosbie and Shutova [10] ablate induction heads in two pretrained models and find that few-shot performance falls by up to about 32% on abstract pattern tasks. Akyürek et al. [11] find that Transformers' success at in-context language learning relies on specialised "n-gram heads", higher-order variants of induction heads. The paper also speculates that one-layer models' different scaling laws may be because they have no induction heads; that is the subject of the next dissection in this section's canon (see scaling laws in the library).

24Reading the paper closely

Where does the paper disagree with itself?

Nine places, none of which changes its argument and all of which a replicator meets.

Two smaller points about the copy this page quotes from. The arXiv e-print of the paper is a 61-page PDF print of the web article rather than LaTeX, so every quote on this page is from that PDF's text layer. And the PDF flattens superscripts, so the small models' vocabulary prints as "216"; the page reads it as 2¹⁶.

25Reading the paper closely

What did later work confirm, refine or overturn?

Singh et al. [7] train a small two-layer model on a synthetic in-context task and clamp parts of it during training. They identify three subcircuits that interact to drive formation of the head, find that heads act additively, and caution that knock-out ablation may hide redundancy; this refines the paper's Argument 3 without contradicting it.

Reddy [9] reports that in-context learning is driven by the abrupt emergence of an induction head in a minimal attention-only network and models the abruptness; this confirms the paper's picture in a controlled setting and gives a mechanism for the plateau. Crosbie and Shutova [10] and Akyürek et al. [11] extend the circuit to pretrained models and to in-context language learning. Bietti et al. [2] study the same setting as this page and get the same order of learning. Abrupt capability changes of this kind are the subject of grokking.

A replication comment in the paper found induction “easily learned with a context length of 4” on a synthetic dataset, which fits this page's 64. Two things the paper left for later are visible in these papers and in the research desk: how many seeds the phase change is stable over, and whether the heads are one circuit or several. The desk below returns to both.

26Research desk

Does the induction head form in a sudden phase change that coincides with in-context learning and a loss bump?

Claim

Argument 1 of Olsson et al. (2022): induction heads "form abruptly during exactly the same window where in-context learning develops", with a loss bump, in every model but the one-layer one induction heads never form. The paper reports one run of training per model.

Hypothesis

H1a: in every seed of the two-layer model the rise is sudden and the three events (induction head, in-context learning, loss descent) coincide. H1b: the paper's “over 75% of final in-context learning forms in this window” holds around the induction head's onset. H1c: a one-layer model never forms an induction head.

Design

Five seeds of the two-layer model, three of the one-layer. 3,000 steps each, snapshots every 20 steps. Verdict rules were written, in numbers, before any run (research/README.md): sudden is a 10% to 90% rise within 600 steps; coincide is a spread of at most 150 steps between the three half-rise steps and a loss bump at least three times steeper than the plateau before it; H1b is supported at a mean share of at least 0.75; H1c is supported if no snapshot of any one-layer seed has a prefix-matching score of 0.10 or more and final accuracy on repeated random sequences is below 20%.

27Research desk

If the architecture can form induction heads without a second layer, does the phase change move?

Claim

Argument 2 predicts that the phase change will happen in one-layer models with this change, and perhaps earlier in models with more layers. The paper reports that in-context learning forms in one-layer models and earlier in deeper ones. The smeared-key model (Model Details) replaces each key by a mixture with its predecessor; the first token's key is unchanged.

Hypothesis

H2a: a one-layer model with smeared keys forms an induction head. H2b: in two layers, smeared keys bring the induction head and the in-context learning score earlier.

Design

Three seeds of one-layer and of two-layer smeared-key models, same data and recipe as E1. H2a is supported if at least 2 of 3 one-layer seeds end with a prefix-matching score of at least 0.30 and accuracy on repeated random sequences of at least 0.60. H2b is supported if the median half-rise of both the induction head and the in-context learning score is at most 0.9 times that of the vanilla two-layer arm.

\tilde{k}_{j} = σ (α) k_{j} + (1 - σ (α)) k_{j - 1}

28Research desk

Do the induction heads, and the previous-token heads they read, carry the in-context learning?

Claim

Argument 3: "almost all the in-context learning in small attention-only models appears to come from these induction heads". And the mechanism: the pattern-preserving ablation of a first-layer head leaves intact the K-composition route, so it should cost little, while ablating the head fully should cost a great deal.

Hypothesis

H3a: pattern-preserving ablation of all layer 2 heads with a prefix-matching score of at least 0.25 removes at least 75% of the in-context learning score that attention provides. H3b: full ablation of the layer 1 heads with a previous-token score of at least 0.15 removes at least 60%, while pattern-preserving ablation of the same heads removes at most 20%.

Design

The final model of each of the five two-layer seeds; ICL score on 1,024 held-out sequences. "Removed share" is (intact − ablated) / (intact − every head ablated), so 100% means the ablation removes everything attention contributes.

29Research desk

What else did the runs show?

Exploratory. Nothing in this step was pre-registered; the observations came from looking at the logs after the verdicts were fixed, and each is a lead, not a result.

30Research desk

What would settle what this page leaves open?

Seeds at scale. (See multi-seed reporting.) The paper reports one run per model (Model Analysis Table). Do its phase-change timings vary between seeds as much as this page's? Training the paper's 2-layer attention-only model from several seeds would answer it directly.
Which heads. Singh et al. [7] find that several heads share the induction job. Does the number of induction heads grow with width, and is the ablation share of E3 still near 100% when the layer has 12 heads?
The confound. The paper's open alternative is that the phase change is the point at which composition is learned. A data set in which composition helps but induction does not (for example, a task needing an unrelated two-head circuit) would separate the two.
Natural text. This page's repeats are verbatim copies of random spans. How do the timings and the head counts change when repeats are fuzzy, as in the paper's Argument 4?
The scaling-law link. The paper suggests one-layer models' different scaling laws may be because they have no induction heads. A grid of one- and two-layer models across widths would test it; that is the subject of the canon's next entry, neural scaling laws.

References

Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S. and Olah, C. (2022). In-context Learning and Induction Heads. Transformer Circuits Thread, 8 March 2022; arXiv:2209.11895v1 The subject of this page. The arXiv e-print is a 61-page PDF of the article, not LaTeX, so every quote on this page comes from that PDF's text layer.
Bietti, A., Cabannes, V., Bouchacourt, D., Jegou, H. and Bottou, L. (2023). Birth of a Transformer: A Memory Viewpoint. NeurIPS 2023; arXiv:2306.00802 Abstract read; quotes are from the abstract.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361 The origin, per Olsson et al., of measuring in-context learning as loss at different token indices.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020; arXiv:2005.14165 The few-shot conception of in-context learning that Olsson et al. set aside in favour of the loss-by-index one.
Press, O., Smith, N. A. and Lewis, M. (2021). Shortformer: Better Language Modeling using Shorter Inputs. ACL 2021; arXiv:2012.15832 Position embeddings added to queries and keys at every layer rather than to the residual stream; the page's miniature uses this reading of Olsson et al.'s 'similar to Press et al.'
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. (2017). Attention Is All You Need. NIPS 2017; arXiv:1706.03762 Scaled dot-product attention. Dissected in 3D at /anatomy/transformer.
Singh, A. K., Moskovitz, T., Hill, F., Chan, S. C. Y. and Saxe, A. M. (2024). What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation. ICML 2024; arXiv:2404.07129 Quotes from the arXiv LaTeX source.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S. and Olah, C. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread The paper that found induction heads in two-layer attention-only models. Cited here only for what Olsson et al. say about it; not fetched for this page.
Reddy, G. (2023). The mechanistic basis of data dependence and abrupt learning in an in-context classification task. ICLR 2024; arXiv:2312.03002 Quotes from the arXiv LaTeX source.
Crosbie, J. and Shutova, E. (2024). Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning. arXiv:2407.07011 Quotes from the arXiv LaTeX source.
Akyürek, E., Wang, B., Kim, Y. and Andreas, J. (2024). In-Context Language Learning: Architectures and Algorithms. ICML 2024; arXiv:2401.12973 Abstract read; quotes are from the abstract.

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
[A][B] … [A] → [B]	the algorithm	[1] Abstract “Induction heads” are attention heads that implement a simple algorithm to complete token sequences like [A][B] … [A] → [B].
induction heads might be the mechanism for the majority of in-context learning	the paper's hypothesis	[1] Abstract we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all
the beginnings of evidence	the paper's own verdict on its evidence	[1] Introduction We emphasize that our results here are only the beginnings of evidence for such a case
1 to 6 layers, 768 wide, 12 heads, 8192 context	the small models	[1] Model Details They have a context window of 8192 tokens, a 216 token vocabulary, an activation dimension dmodel = 768, and 12 attention heads per layer regardless of total model size.
10,000 steps, about 10 billion tokens, 200 snapshots	training of the small models	[1] Model Details They were trained for 10,000 steps (~10 billion tokens), saving 200 snapshots at intervals of every 50 steps.
25 random tokens repeated 4 times behind a start token	the prefix-matching evaluator	[1] Methods · Head activation evaluators Prefix matching: Generate a sequence of 25 random tokens, excluding the most common and the least common tokens. Repeat this sequence 4 times and prepend a “start of sequence” token.
prefix matching and copying, on repeated random tokens	the definition of an induction head	[1] Key Concepts · Induction Heads In other words, induction heads are any heads that empirically increase the likelihood of [B] given [A][B]...[A] when shown a repeated sequence of completely random tokens.
global and context-specific bigrams	a synthetic setup like this page	[2] Abstract We study how transformers balance these two types of knowledge by considering a synthetic setup where tokens are generated from either global or context-specific bigram distributions.
global bigrams fast, the induction head slowly	the order of learning	[2] Abstract we illustrate the fast learning of global bigrams and the slower development of an "induction head" mechanism for the in-context bigrams.
Kaplan et al. 2020	where the loss-by-index definition comes from	[1] Related Work · In-context learning Kaplan et al. [13] is the origin of our approach for using loss at different token indices as a formalism for studying in-context learning.
500th-token loss minus 50th-token loss	in-context learning score	[1] Key Concepts · In-context Learning In-context learning score: the loss of the 500th token in the context minus the average loss of the 50th token in the context, averaged over dataset examples.
10,000 examples of 512 tokens	the per-token loss set	[1] Methods · Per-token losses We run the model snapshot on a set of 10,000 randomly-selected examples from the dataset, each of which is 512 tokens in length
chosen "somewhat arbitrarily"	why tokens 50 and 500	[1] Key Concepts We chose the 500th and 50th token indices somewhat arbitrarily.
50th minus 500th	the other order of the ICL score	[1] Argument 1 if we measure in-context learning for transformer models over the course of training (defined as the 50th token loss minus the 500th token loss as described in Key Concepts), it develops abruptly
< 0.15 nats before, about 0.4 after	in-context learning before and after	[1] Argument 1 Before this window there is less than 0.15 nats of in-context learning, after it there is roughly 0.4 nats, an amount that remains constant for the rest of training and is also constant across many different model sizes
a variant of standard positional embeddings, similar to Press et al.	positions in the small models	[1] Model Details Their positional embeddings are implemented with a variant on the standard positional embeddings (similar to Press et al. [17] ).
these models add none	no position in the residual stream	[1] Argument 5 This mechanism isn't available to the models we study here, since they do not add positional information into the residual stream.
logits are a sum of head terms	why ablation is principled in attention-only models	[1] Footnote 19 In attention-only models, the logits can be expressed (up to a rescaling due to LayerNorm) as a sum of terms from each attention head, along with a "direct path term" to the the token embedding
different circuits, different parameters	attention pattern and output are separable	[1] Footnote 29 the calculation of each attention head’s attention pattern and output vector is separable, that is, implemented by different circuits consisting of different sets of parameters
a previous-token head, then the induction head	the circuit	[1] Introduction Mechanically, induction heads in our models are implemented by a circuit of two attention heads: the first head is a “previous token head” which copies information from the previous token into the next token, while the second head (the actual “induction head”) uses that information to find tokens preceded by the present token.
a continuum	induction heads are not exact	[1] Footnote 5 In practice, induction heads don't exhibit these properties perfectly, and our measurements give us a continuum
with the previous-token head	prefix matching is K-composition	[1] Argument 5 Prefix matching is implemented with K-composition (and to a lesser extent Q-composition) in the QK ("Query-Key") Circuit.
a single head cannot see the token before the one it attends to	why induction needs two layers	[1] Footnote 15 Without using information written by a second earlier attention head, the score from the attended token can’t be a function of the token that precedes it.
composition of heads in different layers	induction heads occur in two-layer but not one-layer models	[1] Footnote 2 Note that induction heads don’t occur in 1 layer models, because they require a composition of attention heads in different layers.
attends back to previous tokens that were followed by the current and/or recent tokens	prefix matching	[1] Key Concepts · Induction Heads Prefix matching: The head attends back to previous tokens that were followed by the current and/or recent tokens.
the head's output increases the logit of the attended-to token	copying	[1] Key Concepts · Induction Heads Copying: The head’s output increases the logit corresponding to the attended-to token.
2022-09-20	the correction to the prefix-matching definition	[1] Change Log 2022-09-20: Corrects the definition of Prefix matching in two places, which incorrectly stated “tokens that were preceded by the current token” instead of “preceded” or “were followed by”.
a ratio of boosts	the copying evaluator	[1] Methods · Head activation evaluators Compute the ratio of the amount it raises the logits of the token being attended to, to that of all tokens in this sample.
a score between -1 and 1	the copying evaluator	[1] Methods · Head activation evaluators This value ranges from 0 (only raises other tokens) to 0.5 (only raises the present token), so we scale it into the range of -1 to 1.
exact previous-token scores recovered 99% of the loss difference	Redwood replication: the previous-token head	[1] Comments & Replications · Adam Scherlis Replacing the previous-token head’s scores with exact previous-token attention recovered 99% of the loss difference.
a simple approximation recovered about 65%	Redwood replication: the induction head	[1] Comments & Replications · Adam Scherlis Replacing the induction head’s scores with a simple approximation (attending to the first token and to exact [A][B]...[A] matches) recovered about 65% of the loss difference.
completing [A][B] … [A] → [B]	induction as nearest neighbour	[1] Introduction the thesis is that there are circuits which have the same or similar mechanism to the 2-layer induction heads and which perform a “fuzzy” or “nearest neighbor” version of pattern completion
the first 1.5e9 tokens	warm-up	[1] Model Details the learning rate warm-up, which ramps up over the first 1.5e9 tokens.
the warm-up falls within the range	the warm-up and the phase change	[1] Model Details · Small models The only other hyperparameter change that occurs within the range of the phase change is the learning rate warm-up
at 4,750 steps	weight-decay change	[1] Model Details Weight decay was reduced at 4750 steps (approximately 5 billion tokens)
three nested logits learned in sequence	a model of the abruptness	[9] Abstract A phenomenological model of induction head formation traces its abrupt emergence to the sequential learning of three nested logits enabled by an intrinsic curriculum.
roughly 2.5 to 5 billion tokens	when in-context learning develops	[1] Argument 1 it develops abruptly in a narrow window early in training (roughly 2.5 to 5 billion tokens) and then is constant for the rest of training
ICL, induction heads, the bump, the trajectory	what co-occurs in the window	[1] Argument 1 Collectively these results suggest that some important transition is happening during the 2.5e9 to 5e9 token window early in training
the only place the loss is not convex	the loss bump	[1] Argument 1 It is in fact the only place in training where the loss is not convex (monotonically decreasing in slope).
approximately 1-3 billion tokens	when the phase change appears in the small models	[1] Model Details · Small models We observe a “phase change” phenomenon that appears at approximately 1-3 billion tokens in the small models.
34 models, more than 50,000 head ablations	models and ablations	[1] Introduction We then present the 6 arguments one by one, drawing on evidence from analysis of 34 transformers over the course of training, including more than 50,000 attention head ablations
strong, causal	strength of the evidence for small attention-only models	[1] Arguments (summary table) For small, attention-only models, we believe we have strong evidence that attention heads are the mechanism for the majority of in-context learning, as we have evidence supported by ablations and mechanistic reverse engineering.
strong and causal for small attention-only models, correlational for larger ones	the abstract's evidence levels	[1] Abstract For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence.
zero the head, force the recorded patterns	pattern-preserving ablation	[1] Methods · Attention head ablations Step 2: Run model a second time. Replace the ablated attention head’s result vector with a zero vector, and force all attention patterns to be the version recorded in the first run.
almost all in-context learning	what ablation shows in small attention-only models	[1] Argument 3 In fact, almost all the in-context learning in small attention-only models appears to come from these induction heads!
multiple heads, none individually necessary	induction heads act additively	[7] §4.2 Furthermore, heads seem to have an additive effect
they may hide redundancy	a caution about knock-out ablations	[7] §4.2 Our analysis builds on this work and cautions against the use of purely knock-out ablations in mechanistic analyses, which may overlook redundancies and incorrectly dismiss network pieces as not implementing a certain function.
footnote 17	ablating most heads raises the score	[1] Footnote 17 In fact, ablating most other attention heads appears to increase in-context learning.
a 40-layer, 13B model	abstract induction heads	[1] Argument 4 we show some anecdotal examples of induction heads from larger transformers (our 40-layer model with 13 billion parameters) that exhibit exactly such behaviors – namely literal copying, translation, and a specific type of abstract pattern matching.
about 65% of its attention on the correct positions	the pattern-matching head	[1] Argument 4 empirically it allocates about 65% of its attention from the colons to the correct positions, when tested on a range of similar problems.
the same sharp increase, with the same rough amounts	continuity from small to large	[1] Argument 6 All of them have the same sharp increase in in-context learning, with the same rough amounts before and after the transition.
4 layers, 13M non-embedding parameters, d_model 512	smallest full-scale model	[1] Model Details · Table of model properties 4 13M 512 8 64
40 layers, 13B non-embedding parameters, d_model 5120	largest full-scale model	[1] Model Details · Table of model properties 40 13B 5120 40 128
d_model = 128 × layers	width of the full-scale models	[1] Model Details · Full-scale models with an activation dimension dmodel = 128 ∗ nlayer
few-shot accuracy falls by up to about 32%	ablating induction heads in pretrained models	[10] Abstract Our results show that even a minimal ablation of induction heads leads to ICL performance decreases of up to ~32% for abstract pattern recognition tasks, bringing the performance close to random.
"n-gram heads"	higher-order induction heads	[11] Abstract we provide evidence that their ability to do so relies on specialized "n-gram heads" (higher-order variants of induction heads) that compute input-conditional next-token distributions.
plausibly different because they have no induction heads	one-layer models and scaling laws	[1] Related Work · Scaling laws It seems quite plausible that the reason why the scaling laws are different for one-layer models is that they don't have induction heads.
by mistake	a start token left out of the copying evaluator	[1] Footnote 31 It would’ve been better if we had used a start of sequence token for the copying head evaluator as well, but we omitted it by mistake.
one	runs per model	[1] Model Analysis Table The arguments above are based on analysis of 34 decoder-only Transformer language models, with different snapshots saved over the course of training, for one run of training per model.
15 points in time	time resolution of the large-model data	[1] Argument 1 · Assessing the Evidence In large models, we have low time resolution on our analysis over training. Co-occurrence when one only has 15 points in time is less surprising and weaker evidence.
about the same for every model after the phase change	a curiosity: the constant score	[1] Unexplained Curiosities is more or less the same for all models after the phase change.
the model may be learning to compose layers	the confound the paper names	[1] Argument 1 · Assessing the Evidence perhaps the phase change is really the point at which the model learns how to compose layers through the residual stream, enabling both induction heads and potentially many other mechanisms that also require composition of multiple heads
none appears to	whether a scheduled change explains the phase change	[1] Argument 1 · Looking at the Phase Change More Closely the window where the phase change happens doesn’t appear to correspond to a scheduled change in learning rate, warmup, or weight decay
three	subcircuits of induction head formation	[7] Abstract By clamping subsets of activations throughout training, we then identify three underlying subcircuits that interact to drive IH formation, yielding the phase change.
of an induction head	in-context learning is driven by an abrupt emergence	[9] Abstract In-context learning (ICL) is driven by the abrupt emergence of an induction head, which subsequently competes with in-weights learning.
on a synthetic dataset	induction learned at context length 4	[1] Comments & Replications · Tom Lieberum On a synthetic dataset, I have already found induction to be easily learned with a context length of 4.
abruptly, in the same window as in-context learning	when induction heads form	[1] Argument 1 we find that induction heads form abruptly during exactly the same window where in-context learning develops
induction heads never form	one-layer models	[1] Argument 1 The exception is the one-layer model, where induction heads never form – just as in-context learning never substantially develops for the one-layer model.
over 75%	share of final in-context learning formed in the window	[1] Argument 1 · Assessing the Evidence Since over 75% of final in-context learning forms in this window, one might naively believe this to be the amount of in-context learning that induction heads are responsible for.
the phase change moves into one-layer models and earlier in deeper ones	the smear experiment's prediction	[1] Argument 2 The hypothesis that induction heads are the primary mechanism of in-context learning predicts that the phase change will happen in one-layer models with this change, and perhaps might happen earlier in models with more layers.
in-context learning forms in one-layer models and earlier in deeper ones	the smear experiment's result	[1] Argument 2 when we use the smeared-key architecture, in-context learning does indeed form for one-layer models (when it didn’t before), and it forms earlier for two-layer and larger models.
σ(α) interpolates between the current and the previous key	smeared keys	[1] Argument 2 · Model Details we introduce a trainable real parameter α used as σ(α) ∈ [0, 1] that interpolates between the key for the current token and previous token
unchanged	smeared keys, first token	[1] Footnote 16 The key vector for the first token is unchanged.

How this was made

Everything on the stage is computed in the page from trained weights: a 45,376-parameter attention-only Transformer trained in PyTorch for 3,000 steps on a CPU. Its JavaScript forward pass matches PyTorch to within 4e-5 on three inputs, an earlier training snapshot, a randomly initialised smeared-key network, and pattern-preserving and full head ablations (test.mjs), and the same tests recompute every derived number in the text: parameter counts, the paper's non-embedding column, the in-context learning score from the stored curves, and the behaviours the text claims on the default input. Paper figures are quoted from the arXiv v1 PDF's text layer and from the follow-up papers' sources, and listed in the fact ledger. The research desk is reproducible from research/: 14 runs across four arms, pre-registered in research/README.md.

Built with the dissection skill. The visual grammar of tensors as blocks of cells owes much to Brendan Bycroft's LLM Visualization. Source, weights, training scripts and experiments: anatomy/ai-ml/2026-10-20-induction-heads/ in the site's repository.

Cite this page

@misc{sigmoid-induction-heads,
  title  = {Anatomy of Induction Heads},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/induction-heads}
}