A complete dissection of Rumelhart, Hinton and Williams (1986), Learning representations by back-propagating errors. The network of the letter's Fig. 3 (24 people and 12 relationships in, codes of 6 and 6, a central layer of 12, a penultimate layer of 6, 24 people out; 630 weights) is laid out on a 3D stage at full size, with the same machine run backwards beside it: for every activation its derivatives, for every weight matrix its gradient, each a block of real numbers computed in the page on the reader's query. A 30-step article follows equations (1) to (9) in the letter's notation, from the one-hot lookup through the logistic unit, the error with its margin, the chain rule, the outer product and the error sent down a layer, to summed sweeps, momentum, weight decay, the receptive fields of Fig. 4, the symmetry network of Fig. 1 and the recurrent nets of Fig. 5, with every number from the letter hover-linked to its sentence. A pre-registered research desk then re-runs four of the letter's single-run claims: whether the stated recipe trains, how often the hidden units rediscover nationality, generation and branch across seeds and widths, whether the network generalises to held-out queries, and whether more hidden units prevent the network getting stuck.
The machine
Two towers on a table. Left, the forward pass of Fig. 3 bottom to top: each layer drawn as a matrix-vector product, the input column beside the weight block with rows aligned and the total input and output above it with columns aligned; a teal thread shows the one-hot lookup, and the target and error sit at the top. Right, the backward pass at the same positions: dE/dy, dE/dx and dE/dw for every layer, red threads carrying the error down, a gold thread carrying it across from the forward tower. In front, plates for Fig. 4 (each person-code unit's 24 weights, English above Italian) and Fig. 1 (the symmetry network on all 64 patterns). Cell colour and height are the values.
What is real. Everything on the stage is computed in the page by engine.js (float64) from weights trained with the same code: the family-trees network (630 parameters, the letter's architecture, accumulated gradients, equation (9), two phases, 0.2% decay, the 0.8/0.2 margin; initial weights ±1, epsilon 0.05 and 5,000 sweeps, because the stated recipe did not train here) and the 6-2-1 symmetry network with the letter's Fig. 1 recipe. test.mjs (181 checks) proves the backward pass against PyTorch autograd for all 630 parameters (to 1e-14) and against finite differences, thirty sweeps of the training rule against PyTorch (1e-15), every output for all 104 queries and 64 symmetry patterns, the family trees against an independent Python derivation (104 queries, 112 triples), exact reproduction of the published weights, and every derived number in the text. Quotes are transcribed from a scan of the printed letter and checked against the page images. The research desk (174 runs plus 38 exploratory continuations) is reproducible from research/.
Working model · 3D
AI & ML · Language models
