A complete, granular dissection of Olsson et al. (2022), In-context Learning and Induction Heads. The paper's small attention-only Transformer is rebuilt at miniature size (two layers, four heads of width 16, 45,376 parameters, positions on queries and keys only) and trained for 3,000 steps on a synthetic language that mixes a Markov chain with repeated random spans. Every tensor of its forward pass on the text you type is drawn on a 3D stage as a block of cells: the embedding table, LayerNorm, queries, keys and values, every head's scores, softmax and output, the residual stream climbing from layer 1 to layer 2, the orange route by which layer 2's keys read what layer 1 wrote, and the unembedding. Thirty steps walk the camera through the computation, then replay training from eighteen snapshots, then through the paper's six arguments, its inconsistencies, and later work, each with the paper's numbers hover-linked to the sentence they come from. A research desk, with hypotheses and numeric verdict rules written before the runs, tests whether the induction head forms in one window with in-context learning and a loss bump (five seeds), whether smeared keys move the phase change, and whether ablations support the mechanism, and reports every seed.
The machine
A staircase on a table. Layer 1 is the lower-left tower, layer 2 the upper-right one, and the residual stream climbs between them. Each tower holds, from the bottom, the residual stream, its LayerNorm, the query/key input with positions added, the three weight matrices with their query, key and value blocks above them, four pods of scores, softmax weights and head outputs, the concatenation, W_O, the attention output and the new residual stream. The embedding and position tables sit at the lower left, the unembedding, logits and probabilities at the top right, and an orange line carries layer 1's output into layer 2's keys. Cell colour and height are the values.
What is real. Everything on the stage is computed in the page from trained weights: a 45,376-parameter attention-only Transformer trained in PyTorch for 3,000 steps on a CPU. Its JavaScript forward pass matches PyTorch to within 4e-5 on three inputs, an earlier training snapshot, a randomly initialised smeared-key network, and pattern-preserving and full head ablations (test.mjs), and the same tests recompute every derived number in the text: parameter counts, the paper's non-embedding column, the in-context learning score from the stored curves, and the behaviours the text claims on the default input. Paper figures are quoted from the arXiv v1 PDF's text layer and from the follow-up papers' sources, and listed in the fact ledger. The research desk is reproducible from research/: 14 runs across four arms, pre-registered in research/README.md.
Working model · 3D
AI & ML · Language models
