Dissection · Rumelhart, Hinton & Williams, 1986

Anatomy of Backpropagation

Rumelhart, Hinton and Williams (1986), dissected: the family-trees network of their Fig. 3 at full size, its forward pass and every derivative of its backward pass drawn as real blocks, checked against PyTorch, with a research desk that re-runs the letter's single-run claims across seeds.

PaperRumelhart, Hinton & Williams, Nature 1986

Model630 parameters, live

Reading26 min · 30 steps

Published2026-10-15

01Overview

A network that learns what to represent

In October 1986 a four-page letter in Nature described "a new learning procedure, back-propagation, for networks of neurone-like units" [1]. The procedure adjusts every weight in a layered network by gradient descent on its error. The claim that made the letter famous was about what came out: hidden units, which nobody tells what to do, "come to represent important features of the task domain".

The letter was received on 1 May 1986 and accepted on 31 July. What you are looking at is its procedure, run on the paper's own network. On the left is the forward pass of the family-trees network of the letter's Fig. 3, bottom to top. On the right is the same machine run backwards: every derivative the procedure computes, drawn as its own block, beside the thing it is the derivative of. In front of them are two exhibits from the paper: the receptive fields of Fig. 4 and the mirror-symmetry network of Fig. 1. Nothing here is a diagram. Every block is a vector or matrix from a network trained in this page's code, and every cell is one number in it: its colour and height are the value. Hover any cell to read it.

The visual grammar, tensors as blocks of cells, follows Brendan Bycroft's LLM Visualization [2]. Scroll, and the camera goes where the text goes.

How to use this page

Scroll the article, or use ← →. Drag the stage to orbit, right-drag to pan, scroll on it to zoom; O shows the whole machine, C returns to the step's view, R replays its animation. Type a query in the bar above, a person and a relationship such as Sophia uncle, and every block is recomputed.

Go deeper: Backpropagation and automatic differentiation

02Overview

Why hidden units were the point

Learning rules for a single layer of weights were old. The perceptron-convergence procedure [3] adjusts the connections from inputs to outputs, and Minsky and Papert had shown how much such networks cannot do [4]. Perceptrons did have intermediate "feature analysers", but, the letter points out, "their input connections are fixed by hand, so their states are completely determined by the input vector: they do not learn representations." The difficulty was the hidden unit: a unit whose correct state is not given by the task, so no one can say whether it is wrong. The letter's answer was that the ability to create useful new features is what back-propagation adds.

The network on the stage is the one in Fig. 3, at the paper's own size. It has 24 input units for person 1 and 12 for the relationship, each group connected to its own group of 6 units, then a central layer of 12 and a penultimate layer of 6, and 24 outputs, one per person. That is weights and biases, few enough that every weight is a cell on the stage (each bias is in the hover text of the unit it belongs to).

Three training settings differ from the paper, and the table shows them in bold. With the training settings the letter states, this network never left its starting plateau in any of 20 attempts here, so the page's model was trained with a larger step and for longer. The research desk at the end reports that test in full. Everything else, from the architecture to the margin on the error, is the paper's.

Go deeper: Calculus and gradients

03The task

Twenty-four people, twelve relationships

The task is to store two family trees, one English and one Italian, with the same shape (Fig. 2). Knowledge is a set of triples, person 1, relationship, person 2, over twelve relationships: father, mother, husband, wife, son, daughter, uncle, aunt, brother, sister, nephew and niece. A network "knows" a triple if it can produce the third term from the first two.

The current query is . The right answer is ; the network says , .

The encoding gives the network nothing to work with. Each person is one input unit of 24 and each relationship one of 12, so Colin is no more similar to Charlotte than to Tomaso. Whatever the network comes to know about generations, nationalities or branches of the family, it must build for itself.

04Forward pass

The first layer is a lookup

The forward pass sets the input units and then computes each layer from the one below, "starting at the bottom and working upwards until the states of the output units are determined". Each unit first forms its total input, a weighted sum of the outputs of the units below it:

x_{j} = i \sum y_{i} w_{ji}

Here $y_{i}$ is the output of unit $i$ in the layer below, $w_{ji}$ the weight on the connection from $i$ to $j$ , and $x_{j}$ the total input of unit $j$ . A bias is an extra input fixed at 1, whose weight is learned like any other. On the stage, each weight matrix sits under the vector it produces, columns aligned, and its input column stands to its left with rows aligned: the matrix-vector product drawn as one.

With a one-hot input the sum has a single non-zero term. Watch the teal thread: the active person unit selects one row of the 24 × 6 weight block, and that row (plus the bias) is the total input of the six person-code units. It is the operation an embedding table performs in every modern language model; the matrix is the person code.

Go deeper: Input embeddings and lookup

05Forward pass

A smooth threshold

Each unit passes its total input through the logistic function:

y_{j} = \frac{1}{1 + e ^{- x_{j}}}

The output is between 0 and 1: near 0 for a large negative input, near 1 for a large positive one, and 0.5 at zero. A threshold unit would jump; this one rises smoothly, and that is the whole trick. A smooth unit has a derivative everywhere, so a small change in any weight makes a small, computable change in the error. Combining the inputs linearly first and applying the nonlinearity after, the letter notes, "greatly simplifies the learning procedure"; and the exact function does not matter: "Any input-output function which has a bounded derivative will do." Its derivative takes a convenient form, $d y_{j} / d x_{j} = y_{j} (1 - y_{j})$ , which the backward pass will use.

For this query the six person-code units output .

Go deeper: Activation functions · Logistic regression and the log-odds

06Forward pass

The same, for the relationship

The relationship goes through its own 12 × 6 block in the same way: one active unit picks one row. The two groups are kept apart here on purpose. The person code can only see who person 1 is, and the relationship code can only see the relationship, so each is forced to describe its own input in a way that will be useful to the layers above. These groups learn to encode people and relationships as distributed patterns of activity.

A distributed pattern means that each person is described by six real numbers, and each number takes part in describing every person. That is the opposite of the input's one-unit-per-person code, and it is what allows what the network learns about one person to transfer to another.

07Forward pass

Where person and relationship meet

The central layer's 12 units read both codes at once: its input column is the six person-code outputs stacked on the six relationship-code outputs. The animation shows one cell first: the highlighted column of weights, multiplied element by element with the input column and summed, gives one unit's total input. Then the other eleven follow, and the logistic turns each into an output.

The letter's connectivity rule allows more than this: "Connections within a layer or from higher to lower layers are forbidden, but connections can skip intermediate layers." Fig. 3's network uses no skips; each layer reads only the one below it.

08Forward pass

Six numbers to describe the answer

The penultimate layer squeezes the 12 central units into 6. It is the mirror of the person code at the bottom: where those six units describe person 1, these six must describe person 2 well enough that the output layer can name them. The activity in the penultimate layer must activate the correct output units.

09Forward pass

Twenty-four yes-or-no answers

The output layer has one unit per person, laid out like the input with the English and their Italian twins side by side. A unit that comes on (above 0.5) is the network saying "yes, this person". A query may have more than one answer, so several outputs may be on at once. The letter's Fig. 3 shows exactly this query: Colin and "has-aunt", and two black dots for his two aunts.

For the three strongest outputs are .

10Forward pass

How wrong, measured

The procedure minimises a single number, the total error over all cases $c$ and output units $j$ :

E = \frac{1}{2} c \sum j \sum (y_{j, c} - d_{j, c})^{2}

where $d$ is the desired state: 1 for each right answer, 0 for everyone else. The block marked d is this query's target, and y − d the difference. For the family trees the authors added a margin: the error was zero for an output that should be on and was above 0.8, or should be off and was below 0.2, "to prevent the network needing large weights to drive the outputs to 1 or 0". Cells inside the margin are drawn as masked.

For this query, of the 24 outputs are inside the margin and contribute nothing; the error is with the margin and without it.

11Backward pass

The backward pass starts with the error itself

To descend the error, the procedure needs $\partial E / \partial w$ for every weight. For each case it computes them in two passes: the forward pass we have seen, then a backward pass "which propagates derivatives from the top layer back to the bottom one". The gold thread on the stage carries the error across to the backward tower, where it starts at the top.

For one case, differentiating the error with respect to an output gives the paper's equation (4):

\frac{\partial E}{\partial y _{j}} = y_{j} - d_{j}

It is the y − d block, copied: positive where an output is too high, negative where it is too low, zero inside the margin. Live: .

12Backward pass

Through the unit, by the chain rule

Next, how the error changes with each output unit's total input. The chain rule multiplies the error's sensitivity to $y_{j}$ by $y_{j}$ 's sensitivity to $x_{j}$ , which for the logistic is $y_{j} (1 - y_{j})$ , equation (5):

\frac{\partial E}{\partial x _{j}} = \frac{\partial E}{\partial y _{j}} y_{j} (1 - y_{j})

Compare the two rows on the stage. An output near 0 or near 1 has a slope near zero, so even a large error there produces a small $\partial E / \partial x$ : a saturated unit hardly learns. The margin keeps the network from pushing its outputs into that region for no gain.

13Backward pass

Every weight's share of the blame

The total input $x_{j}$ is a linear function of the weights into it, so its derivative with respect to $w_{ji}$ is the input $y_{i}$ . Equation (6):

\frac{\partial E}{\partial w _{ji}} = \frac{\partial E}{\partial x _{j}} y_{i}

A weight's gradient is the error signal at the unit it feeds, times the activity of the unit it comes from. Over a whole layer this is an outer product: the input column times the $\partial E / \partial x$ row. The animation shows one cell first, then fills the block. Its shape is the weight block's shape, and it sits in the backward tower exactly where that weight block sits in the forward one. A weight from a unit that was silent ( $y_{i} = 0$ ) gets no gradient at all.

14Backward pass

The step that makes hidden units learnable

Now the step that the letter is about. A hidden unit $i$ in the layer below has no target, but it affects the error through every unit $j$ it feeds, each by $w_{ji}$ . Summing those effects gives equation (7):

\frac{\partial E}{\partial y _{i}} = j \sum \frac{\partial E}{\partial x _{j}} w_{ji}

On the stage this is the red column beside the output block: the $\partial E / \partial x$ row multiplied by the weight matrix, the same weights as the forward pass, used in the other direction. The result is a $\partial E / \partial y$ for each penultimate unit, a target-free answer to the question "how should this unit have behaved?". The red thread carries it down to the penultimate station, where equations (5) and (6) apply again.

15Backward pass

Repeat until the inputs

With $\partial E / \partial y$ for the penultimate layer the same three steps repeat: through the logistic, an outer product for the weights, a weighted sum to send the error down. "We can therefore repeat this procedure to compute this term for successively earlier layers, computing ∂E/∂w for the weights as we go." At the central layer the error column splits: its top six entries go to the person code and its bottom six to the relationship code.

Every pass through a logistic multiplies the signal by at most 1/4, and every pass through a weight matrix by roughly the size of its weights. In the trained network the weights have grown large enough to balance the slopes: the typical size of $\partial E / \partial x$ for this query, from the output down, is . At the start of training it is different. With weights drawn from ±0.3, the signal reaching the person code is times weaker than at the output, and with weights from ±1 it is times weaker. The bottom layers of a small-weight network barely hear the error, which is one reason training starts on a long plateau, and in deep sigmoid networks the same arithmetic made early layers learn very slowly until better initialisation and momentum schedules arrived [5].

The backward pass costs a small multiple of the forward pass: the same matrices read in the other direction, plus an outer product per layer.

Go deeper: Signal propagation and initialisation

16Backward pass

At the bottom, learning is local

At the first layer the input units are one-hot, so equation (6) gives a non-zero gradient only on the row of the active input. For this query, entries of the person-code gradient are non-zero: the six in row . One case teaches the network about one person's code and nobody else's.

That is why the network has to share structure to generalise. The only path from what it learns about Colin to what it believes about Alfonso is through the shared weights above the codes; the codes themselves are separate rows. The letter's account of Fig. 4 says the network uses exactly this route: "The network is making use of the isomorphism between the two family trees to allow it to share structure".

17Backward pass

Is the gradient right?

A derivative can always be estimated by brute force: nudge one weight up and down by a small $h$ and measure the change in the error. That needs two forward passes per weight. Backpropagation gets every derivative from one forward and one backward pass, and it is exact, not an estimate. The figure compares the two on this query's six largest gradients.

Before this page was published, its JavaScript backward pass was checked against two independent references. The first is PyTorch's automatic differentiation [6], for all 630 weights and biases at two points in training, agreeing to $1 0^{- 14}$ . The second is central finite differences on 90 weights. Thirty sweeps of the full training rule also match PyTorch to $1 0^{- 15}$ . PyTorch's autograd is reverse-mode automatic differentiation: the general method, of which this letter's procedure is the case for layered networks [7].

Go deeper: Backpropagation and automatic differentiation

18Training

Sum over all cases, then change

The gradient so far is for one query. The error is a sum over cases, so its gradient is the sum of the per-case gradients. The authors could have changed the weights after every case, which "has the advantage that no separate memory is required for the derivatives". Instead, they accumulated ∂E/∂w over all the input-output cases before changing the weights. One pass through all the cases is a sweep, and the simplest rule is to step against the summed gradient, equation (8):

Δ w = - ε \frac{\partial E}{\partial w}

Halfway through this step's animation the gradient blocks switch from this one query to the sum over all 100 training queries at the trained weights. The single-query blocks are sparse at the bottom; the summed ones are dense, because between them the queries touch every person and every relationship. At a trained network the sum is small, but it is not zero, because weight decay keeps pulling the other way.

Go deeper: Gradient accumulation

19Training

Let the gradient change the velocity

Plain gradient descent "does not converge as rapidly as methods which make use of the second derivatives, but it is much simpler and can easily be implemented by local computations in parallel hardware". The letter improves it "by using an acceleration method in which the current gradient is used to modify the velocity of the point in weight space instead of its position", equation (9):

Δ w (t) = - ε \frac{\partial E}{\partial w} (t) + α Δ w (t - 1)

$t$ counts sweeps, and α, between 0 and 1, sets how much earlier gradients contribute. Today this is called momentum. It is the heavy-ball method that Polyak analysed for optimisation in 1964 [8], which the letter does not cite. Where successive gradients agree, steps add up to $1/ (1 - α)$ times a single step, ten times for α = 0.9; where they alternate, they cancel.

Momentum turned out to matter far more than its single paragraph suggests: decades later, a carefully tuned momentum schedule and a sensible initialisation were enough to train deep and recurrent networks that had been thought to need second-order methods [5].

Go deeper: Optimisers: Adam, AdamW, Lion · Optimisation theory

20Training

Five thousand sweeps, watched

The letter's recipe for the family trees: 1,500 sweeps, ε = 0.005 and α = 0.5 for the first 20, then ε = 0.01 and α = 0.9. On top of that, weight decay: every weight reduced by 0.2% after each change, introduced "to make it easier to interpret the weights". After prolonged learning the decay is balanced by ∂E/∂w, so a weight's final size says how useful it is. Initial weights must be random, to break symmetry: units that start identical receive identical gradients and stay identical.

The stage replays this page's training run. The weight blocks and the Fig. 4 fields show their values at sweeps 0, 20, 100, 250 and on to 5,000, and the callout counts the training queries the network gets right at each. The run's history:

The network fits all 100 training queries from sweep . The long flat stretch before that is the plateau the research desk examines. The page's model is seed , chosen by a rule fixed in advance: the first seed, counting from 1, that fits the training set.

Go deeper: Weight decay · Learning-rate schedules

21What it learned

Features nobody put there

Fig. 4 of the letter shows the weights from the 24 person units to the 6 person-code units, each unit's weights in two rows, English above their Italian twins. Its caption reads them: "Unit 1 is primarily concerned with the distinction between English and Italian", and "Unit 2 encodes which generation a person belongs to, and unit 6 encodes which branch of the family they come from." None of this is in the encoding, which gives each person a separate unit.

The plate in front of the stage and the figure below are the same picture for this page's network. Scoring each unit by how well its 24 weights correlate with each feature, this network has . The unit numbers differ from the paper's; the order of hidden units is arbitrary. Move the slider to see the same figure for the 16 networks of the research desk.

If the network shares structure between the trees, a person's code should be nearer their twin's than other people's. Try other people: in this network twins are usually somewhat nearer than average, not dramatically so, and how much of the gap sits on the nationality unit varies from person to person.

Go deeper: Knowledge-graph embeddings · Superposition and polysemantic neurons

22What it learned

Does it know what it was not told?

The letter's test of whether the features are real: "the network generalizes correctly to the four triples on which it was not trained". It does not say which four.

This page's network held out four queries and gets right: . Type one into the query bar to see the whole forward pass on it.

Four queries is a small test, and the research desk repeats it over 16 networks with different held-out sets.

Go deeper: Grokking and delayed generalisation

23Fig. 1

A task no single layer can do

The letter's first example is smaller. Six binary inputs; the output should be on when the pattern is a mirror image of itself. An intermediate layer is essential "because the activity in an individual input unit, considered alone, provides no evidence about the symmetry or non-symmetry of the whole input vector". The procedure found a solution with two intermediate units, after 1,425 sweeps through the 64 patterns with ε = 0.1 and α = 0.9 and initial weights uniform in ±0.3.

The solution is elegant. For each hidden unit, weights symmetric about the middle are equal and opposite, so a mirror pattern sums to zero and leaves both hidden units off, and the output, with its positive bias, comes on. The weights on each side are in the ratio 1 : 2 : 4, so each of the eight half-patterns sends a unique sum and only its mirror can cancel it. The letter's figure shows 14.2, 7.2 and 3.6 on one side.

The plate on the stage is the same network, trained here with the paper's settings. It solved the task at sweep ; its first hidden unit's weights are , magnitudes in the ratio . The same structure, with the three pairs in a different order. Train it yourself with any seed:

24Fig. 5

A recurrent net is a layered net, unrolled

The letter's Fig. 5 extends the procedure to networks with loops. A synchronous recurrent net run for three steps is equivalent to a layered net of three layers: "Each time-step in the recurrent net corresponds to a layer in the layered net." Two complications follow. The backward pass needs every layer's outputs, so the recurrent net must store the history of its states. And the copies of a weight in different layers must stay equal, so the procedure averages ∂E/∂w over each set of corresponding weights and changes them together.

On the stage, read the forward tower's central, penultimate and output layers as three time-steps of one recurrent layer, and the backward tower as the error flowing back through time. This is backpropagation through time, still the way recurrent networks are trained; the letter points to its long version for the details [11].

Go deeper: Recurrent networks, LSTM and GRU

25Limits and history

What the letter conceded

Local minima. "The most obvious drawback of the learning procedure is that the error-surface may contain local minima so that gradient descent is not guaranteed to find a global minimum." The authors' experience was reassuring: the network "very rarely gets stuck in poor local minima", and only in networks with barely enough connections for the task, because extra connections create dimensions that lead around the barriers. Later analysis of large networks moved the worry from local minima to saddle points and the plateaus around them [12]. The research desk tests the letter's claim on its own symmetry task.

Brains. "The learning procedure, in its current form, is not a plausible model of learning in brains." The letter ends by suggesting it is worth looking for more biologically plausible ways of doing gradient descent. Thirty-four years later, a review co-authored by Hinton argued that cortex could implement the core of the procedure, with feedback connections inducing activity differences that carry error signals [13].

Priority. The letter notes that variants were discovered independently by David Parker and by Yann Le Cun [14]. It does not mention earlier work. The general method, reverse-mode differentiation of a computation, appeared in Linnainmaa's 1970 master's thesis [15]; Werbos described training neural networks this way in his 1974 thesis [16] and applied it to networks in 1982 [17]. Surveys of the history give these in detail [18]. What the 1986 letter added was the demonstration that mattered to the field: hidden units learning features worth reading.

Go deeper: Loss landscapes · Boltzmann machines and hidden units

26Research desk

Reproducing the recipe as printed

The research desk tests four things the letter reported from single runs. Hypotheses and verdict rules were written down before any of the runs below, in research/README.md; the result paragraphs on this page are generated from the result files by the same rules, so they cannot be edited to fit. Every family-trees run uses the network on the stage, with 4 held-out queries drawn per seed.

Claim

"We trained the network for 1500 sweeps, using ε = 0.005 and α = 0.5 for the first 20 sweeps and ε = 0.01 and α = 0.9 for the remaining sweeps."

Hypothesis

In this implementation, that recipe fits all 100 training queries in fewer than 2 of 10 seeds. This is a confirmation: pilot runs while building the page had already shown it failing.

Design

The recipe exactly, with decay and margin, 1,500 sweeps, 10 seeds with initial weights in ±0.3 (Fig. 1's range) and 10 in ±1. Supported if at most 1 of 10 fits in both arms.

What the plateau looks like inside: in seed 1 of each arm, no output on any of the 100 training queries rises above 0.21. The network has parked every output at the lower edge of the margin, where an output that should be off stops costing anything, and the 107 outputs that should be on across the training set pay the whole error. On the stage, this page's trained network shows the way out: for Colin aunt the two right answers have climbed to about 0.8 and everyone else sits at or below about 0.2.

27Research desk

Nationality, generation, branch: how often?

Claim

Fig. 4 shows one network whose person code has a nationality unit and a generation unit and a branch unit. How often training produces them is not reported.

Hypothesis

At the paper's width of 6 units, each feature appears in at least 70% of seeds; fewer units find them less often (at least 10 points lower at 3 units), more units no less often; and without weight decay the features are at least 15 points rarer.

Design

Widths 3, 6 and 12, and 6 without decay; 16 seeds each, 5,000 sweeps of this page's recipe. A feature is found when some unit's 24 weights correlate with it at |r| ≥ 0.75. Branch is scored on the 20 people who belong to one side of the family.

28Research desk

Four held-out queries, sixteen times

Claim

"the network generalizes correctly to the four triples on which it was not trained", because the hidden features capture the underlying structure of the task.

Hypothesis

Averaged over seeds, the network answers at least 3 of its 4 held-out queries correctly.

Design

The 16 width-6 networks of E2, each with its own random 4 held-out queries. A query is right when every output is on the correct side of 0.5. Supported if the mean is ≥ 3, partly if 2 to 3, not if below 2.

Queries differ in how much the rest of the data implies their answer, so four of them can make generalisation look easy or hard. The letter does not say which four it held out, or how many runs it tried, so whether its 4 of 4 was typical cannot be checked from the letter.

29Research desk

Do extra connections prevent getting stuck?

Claim

"We have only encountered this undesirable behaviour in networks that have just enough connections to perform the task." Extra connections provide "paths around the barriers". Not quantified.

Hypothesis

On the symmetry task with 2 hidden units, the minimum the paper's solution uses, some runs fail; with 4 hidden units, at least 15 points fewer fail.

Design

Fig. 1's recipe exactly (ε = 0.1, α = 0.9, weights from ±0.3), 2, 3 or 4 hidden units, 30 seeds each, up to 5,000 sweeps. A run fails if any of the 64 patterns is wrong at the end.

You can reproduce single runs above, in the Fig. 1 figure: pick a seed and a number of hidden units.

30Research desk

What this adds, and what to test next

In short: the procedure and its equations hold exactly. The family-trees features are real and appear in most runs, more often in wider codes and with weight decay. The stated training recipe does not train this network, the generalisation the letter reports is the exception rather than the rule here, and what looked like local minima on the symmetry task are mostly plateaus that more sweeps escape. All of it is at the paper's own tiny scale, which is the scale its claims were made at.

What was the 1986 recipe, exactly? Search the unstated settings systematically (initial range, whether biases decay, how the margin enters, online versus summed updates) for the combination under which ε = 0.01 trains in 1,500 sweeps. Hinton's fuller report of the experiment [10] is the place to start.
Which held-out queries can be generalised at all? Classify the 104 queries by whether their answer follows from others (an uncle is a parent's brother) and test held-out sets of each kind separately.
Plateau length against width. E4's stalls get longer, not shorter, with more hidden units. Measure the time to escape against initial weight scale and width, and compare with saddle-point accounts [12].
Do the features cause the answers? Clamp the nationality unit to its Italian value on an English query and check whether the network answers with the Italian twin. A feature that can be steered this way is used, not only present.
Momentum schedules. Re-run E1 with momentum raised slowly from 0.5 to 0.9 over hundreds of sweeps, the schedule later found to matter for deep networks [5].

The engine, the trained weights, the PyTorch reference and every experiment are in this page's directory, anatomy/ai-ml/2026-10-15-backpropagation/; the JavaScript that draws the stage is the code the tests check.

References

Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533–536 (9 October 1986); doi:10.1038/323533a0 The subject of this page. Quotes are transcribed from a scan of the printed letter, checked word by word against the page images.
Bycroft, B. (2023). LLM Visualization. bbycroft.net/llm The visual grammar of tensors as blocks of cells.
Rosenblatt, F. (1961). Principles of Neurodynamics. Spartan, Washington, DC The letter's ref. 1: the perceptron-convergence procedure. Not read for this page.
Minsky, M. L. and Papert, S. (1969). Perceptrons. MIT Press, Cambridge, MA The letter's ref. 2, cited for the proof that symmetry needs intermediate units. Not read for this page.
Sutskever, I., Martens, J., Dahl, G. and Hinton, G. (2013). On the importance of initialization and momentum in deep learning. ICML 2013, PMLR 28, 1139–1147 Read from its abstract: with a careful initialisation and a slowly increasing momentum schedule, plain momentum trains deep and recurrent networks that were thought to need second-order methods, and both ingredients are needed.
Paszke, A. et al. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS 2019; arXiv:1912.01703 The autograd library this page's backward pass is checked against (torch 2.14, CPU, float64).
Baydin, A. G., Pearlmutter, B. A., Radul, A. A. and Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research 18(153), 1–43; arXiv:1502.05767 Read from its abstract and excerpts: backpropagation is reverse-mode automatic differentiation applied to a network.
Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4(5), 1–17 The heavy-ball method, the same update as the letter's equation (9). Known here through its record and secondary descriptions.
Paccanaro, A. and Hinton, G. E. (2001). Learning distributed representations of concepts using Linear Relational Embedding. IEEE Transactions on Knowledge and Data Engineering 13(2), 232–245 Returns to the family trees with relations as matrices acting on concept vectors; read from the technical report's abstract.
Hinton, G. E. (1986). Learning distributed representations of concepts. Proceedings of the Eighth Annual Conference of the Cognitive Science Society, Amherst, MA, 1–12 The fuller report of the family-trees experiment. Only its abstract and secondary descriptions could be read from the build environment.
Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning internal representations by error propagation. In Parallel Distributed Processing, Vol. 1: Foundations (eds Rumelhart & McClelland), 318–362, MIT Press The long version, cited by the letter as its ref. 4. Not read for this page; cited only for what the letter says it contains.
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S. and Bengio, Y. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. NIPS 2014; arXiv:1406.2572 Read from its abstract: in high dimensions saddle points surrounded by plateaus, not poor local minima, are the main obstacle to gradient descent.
Lillicrap, T. P., Santoro, A., Marris, L., Akerman, C. J. and Hinton, G. (2020). Backpropagation and the brain. Nature Reviews Neuroscience 21, 335–346 Read from its abstract: argues the cortex could implement the core of backpropagation by using feedback connections to induce activity differences that carry error signals.
Le Cun, Y. (1985). Une procédure d'apprentissage pour réseau à seuil asymétrique (A learning scheme for asymmetric threshold networks). Proceedings of Cognitiva 85, Paris, 599–604 Cited by the letter as an independent discovery. Bibliographic record only.
Linnainmaa, S. (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's thesis (in Finnish), University of Helsinki; published in English as 'Taylor expansion of the accumulated rounding error', BIT 16, 146–160 (1976) Reverse-mode automatic differentiation, of which backpropagation is the special case for layered networks. Known here through bibliographic records and Griewank's history, not the thesis itself.
Werbos, P. J. (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University; reprinted in The Roots of Backpropagation (Wiley, 1994) Usually credited as the first description of training neural networks by back-propagating errors. Known here through bibliographic records.
Werbos, P. J. (1982). Applications of advances in nonlinear sensitivity analysis. System Modeling and Optimization, Lecture Notes in Control and Information Sciences 38, Springer, 762–770 Werbos's application of reverse-mode differentiation to neural networks. Bibliographic record only.
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks 61, 85–117; arXiv:1404.7828 Includes a history of backpropagation (its §5.5); known here through its abstract and contents.

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
back-propagation	what the paper describes	[1] p. 533, abstract We describe a new learning procedure, back-propagation, for networks of neurone-like units.
hidden units come to represent features	the claim about hidden units	[1] p. 533, abstract internal 'hidden' units which are not part of the input or output come to represent important features of the task domain, and the regularities in the task are captured by the interactions of these units.
received 1 May, accepted 31 July 1986	dates	[1] p. 536 Received 1 May; accepted 31 July 1986.
fixed by hand	perceptron feature analysers	[1] p. 533 their input connections are fixed by hand, so their states are completely determined by the input vector: they do not learn representations.
useful new features	what is new	[1] p. 533, abstract The ability to create useful new features distinguishes back-propagation from earlier, simpler methods such as the perceptron-convergence procedure.
24 + 12	input units	[1] Fig. 3 caption The bottom layer has 24 input units on the left for representing (person 1) and 12 input units on the right for representing the relationship.
6 + 6 units	second layer	[1] Fig. 3 caption Each of the two input groups is totally connected to its own group of 6 units in the second layer. These groups learn to encode people and relationships as distributed patterns of activity.
12 and 6 units	central and penultimate layers	[1] Fig. 3 caption The second layer is totally connected to the central layer of 12 units, and these are connected to the penultimate layer of 6 units.
12	the twelve relationships	[1] Fig. 2 caption where the possible relationships are {father, mother, husband, wife, son, daughter, uncle, aunt, brother, sister, nephew, niece}.
complete the triple	what 'knowing' means	[1] Fig. 2 caption A layered net can be said to 'know' these triples if it can produce the third term of each triple when given the first two.
100 of 104	training queries	[1] p. 535 Fig. 4 shows the 'receptive fields' of some of the hidden units after the network was trained on 100 of the 104 possible triples.
2	two answers	[1] Fig. 3 caption In this case, there are two correct answers (marked by black dots) because Colin has two aunts.
layer by layer, bottom up	order of the forward pass	[1] p. 533 All units within a layer have their states set in parallel, but different layers have their states set sequentially, starting at the bottom and working upwards until the states of the output units are determined.
an extra input fixed at 1	biases	[1] p. 533 Units can be given biases by introducing an extra input to each unit which always has a value of 1. The weight on this extra input is called the bias and is equivalent to a threshold of the opposite sign. It can be treated just like the other weights.
simplifies learning	why a linear sum first	[1] p. 534 the use of a linear function for combining the inputs to a unit before applying the nonlinearity greatly simplifies the learning procedure.
any with a bounded derivative	which nonlinearity	[1] p. 534 It is not necessary to use exactly the functions given in equations (1) and (2). Any input-output function which has a bounded derivative will do.
bottom-up, may skip layers	the connectivity allowed	[1] p. 533 Connections within a layer or from higher to lower layers are forbidden, but connections can skip intermediate layers.
one per person 2	output units	[1] Fig. 3 caption The activity in the penultimate layer must activate the correct output units, each of which stands for a particular (person 2).
Colin, has-aunt	the case shown	[1] Fig. 3 caption There is one active unit in the first group representing Colin and one in the second group representing the relationship 'has-aunt'.
0.8 / 0.2	the error margin	[1] Fig. 4 caption To prevent the network needing large weights to drive the outputs to 1 or 0, the error was considered to be zero if output units that should be on had activities above 0.8 and output units that should be off had activities below 0.2.
two passes	how a gradient is computed	[1] p. 534 For a given case, the partial derivatives of the error with respect to each weight are computed in two passes.
top to bottom	the backward pass	[1] p. 534 The backward pass which propagates derivatives from the top layer back to the bottom one is more complicated.
layer by layer	recursion	[1] p. 535 We can therefore repeat this procedure to compute this term for successively earlier layers, computing ∂E/∂w for the weights as we go.
the two trees	sharing structure	[1] Fig. 4 caption The network is making use of the isomorphism between the two family trees to allow it to share structure and it will therefore tend to generalize sensibly from one tree to the other.
no extra memory	changing after every case	[1] p. 535 One way of using ∂E/∂w is to change the weights after every input-output case. This has the advantage that no separate memory is required for the derivatives.
accumulate over all cases	what the paper did	[1] p. 535 An alternative scheme, which we used in the research reported here, is to accumulate ∂E/∂w over all the input-output cases before changing the weights.
simpler, local	against second-order methods	[1] p. 535 This method does not converge as rapidly as methods which make use of the second derivatives, but it is much simpler and can easily be implemented by local computations in parallel hardware.
gradient changes velocity	the acceleration method	[1] p. 535 by using an acceleration method in which the current gradient is used to modify the velocity of the point in weight space instead of its position
one per sweep	t in equation (9)	[1] p. 535, eq. (9) where t is incremented by 1 for each sweep through the whole set of input-output cases
between 0 and 1	α	[1] p. 535, eq. (9) α is an exponential decay factor between 0 and 1 that determines the relative contribution of the current gradient and earlier gradients to the weight change.
1,500 sweeps; ε 0.005 → 0.01; α 0.5 → 0.9	family-trees recipe	[1] Fig. 4 caption We trained the network for 1500 sweeps, using ε = 0.005 and α = 0.5 for the first 20 sweeps and ε = 0.01 and α = 0.9 for the remaining sweeps.
0.2% per change	weight decay	[1] Fig. 4 caption To make it easier to interpret the weights we introduced 'weight-decay' by decrementing every weight by 0.2% after each weight change.
magnitude = usefulness	what decay buys	[1] Fig. 4 caption After prolonged learning, the decay was balanced by ∂E/∂w, so the final magnitude of each weight indicates its usefulness in reducing the error.
small random weights	initialisation	[1] p. 535 To break symmetry we start with small random weights.
24 × 6 weights	what Fig. 4 shows	[1] Fig. 4 caption The weights from the 24 input units that represent people to the 6 units in the second layer that learn distributed representations of people.
English vs Italian	unit 1	[1] Fig. 4 caption Unit 1 is primarily concerned with the distinction between English and Italian and most of the other units ignore this distinction.
generation, branch	units 2 and 6	[1] Fig. 4 caption Unit 2 encodes which generation a person belongs to, and unit 6 encodes which branch of the family they come from.
one unit per person	features not in the encoding	[1] Fig. 4 caption The features captured by the hidden units are not at all explicit in the input and output encodings, since these use a separate unit for each person.
4 of 4	held-out queries	[1] Fig. 4 caption Because the hidden features capture the underlying structure of the task domain, the network generalizes correctly to the four triples on which it was not trained.
no single input is evidence	why symmetry needs hidden units	[1] p. 535 it is essential to use an intermediate layer because the activity in an individual input unit, considered alone, provides no evidence about the symmetry or non-symmetry of the whole input vector
two intermediate units	the symmetry solution	[1] p. 535 The learning procedure discovered an elegant solution using just two intermediate units, as shown in Fig. 1.
1,425	sweeps to learn symmetry	[1] Fig. 1 caption The learning required 1,425 sweeps through the set of 64 possible input vectors, with the weights being adjusted on the basis of the accumulated gradient after each sweep.
ε = 0.1, α = 0.9	symmetry ε and α	[1] Fig. 1 caption The values of the parameters in equation (9) were ε = 0.1 and α = 0.9.
uniform in ±0.3	symmetry initial weights	[1] Fig. 1 caption The initial weights were random and were uniformly distributed between −0.3 and 0.3.
equal and opposite about the middle	the key property	[1] Fig. 1 caption for a given hidden unit, weights that are symmetric about the middle of the input vector are equal in magnitude and opposite in sign.
1 : 2 : 4	weight ratio	[1] Fig. 1 caption Note that the weights on each side of the midpoint are in the ratio 1:2:4.
a unique sum per pattern	why 1 : 2 : 4	[1] Fig. 1 caption This ensures that each of the eight patterns that can occur above the midpoint sends a unique activation sum to each hidden unit, so the only pattern below the midpoint that can exactly balance this sum is the symmetrical one.
14.2, 7.2, 3.6 (and 7.1)	the drawn weights	[1] Fig. 1 14.2 −3.6 7.2 −7.2 3.6 −14.2 (left hidden unit); −14.2 3.6 −7.1 7.1 −3.6 14.2 (right hidden unit); hidden biases −1.1, output bias 6.4, hidden-to-output weights −8.8
mirror images	the two hidden units	[1] Fig. 1 caption The two hidden units have identical patterns of weights but with opposite signs, so for every non-symmetric pattern one hidden unit will come on and suppress the output unit.
one layer per time-step	recurrent = layered	[1] Fig. 5 caption Each time-step in the recurrent net corresponds to a layer in the layered net.
store the history	first complication	[1] Fig. 5 caption So in an iterative net it is necessary to store the history of output states of each unit.
average tied gradients	second complication	[1] Fig. 5 caption we average ∂E/∂w for all the weights in each set of corresponding weights and then change each weight in the set by an amount proportional to this average gradient.
local minima	the drawback	[1] p. 535 The most obvious drawback of the learning procedure is that the error-surface may contain local minima so that gradient descent is not guaranteed to find a global minimum.
very rarely stuck	experience	[1] p. 535 experience with many tasks shows that the network very rarely gets stuck in poor local minima that are significantly worse than the global minimum.
just enough connections	when it gets stuck	[1] p. 535 We have only encountered this undesirable behaviour in networks that have just enough connections to perform the task.
paths around barriers	why more connections help	[1] p. 535 Adding a few more connections creates extra dimensions in weight-space and these dimensions provide paths around the barriers that create poor local minima in the lower dimensional subspaces.
not plausible	biology	[1] p. 536 The learning procedure, in its current form, is not a plausible model of learning in brains.
look for plausible variants	the closing suggestion	[1] p. 536 this suggests that it is worth looking for more biologically plausible ways of doing gradient descent in neural networks.
Parker; Le Cun	independent discoveries	[1] p. 535 Variants on the learning procedure have been discovered independently by David Parker (personal communication) and by Yann Le Cun
Pittsburgh, Philadelphia	Hinton's affiliation as printed	[1] p. 533, affiliations Department of Computer Science, Carnegie-Mellon University, Pittsburgh, Philadelphia 15213, USA
five	layers	[1] Fig. 3 caption Activity levels in a five-layer network after it has learned.

How this was made

Everything on the stage is computed in the page by engine.js (float64) from weights trained with the same code: the family-trees network (630 parameters, the letter's architecture, accumulated gradients, equation (9), two phases, 0.2% decay, the 0.8/0.2 margin; initial weights ±1, epsilon 0.05 and 5,000 sweeps, because the stated recipe did not train here) and the 6-2-1 symmetry network with the letter's Fig. 1 recipe. test.mjs (181 checks) proves the backward pass against PyTorch autograd for all 630 parameters (to 1e-14) and against finite differences, thirty sweeps of the training rule against PyTorch (1e-15), every output for all 104 queries and 64 symmetry patterns, the family trees against an independent Python derivation (104 queries, 112 triples), exact reproduction of the published weights, and every derived number in the text. Quotes are transcribed from a scan of the printed letter and checked against the page images. The research desk (174 runs plus 38 exploratory continuations) is reproducible from research/.

Built with the dissection skill. The visual grammar of tensors as blocks of cells owes much to Brendan Bycroft's LLM Visualization. The backward pass is checked against PyTorch autograd. Source, weights, training scripts and experiments: anatomy/ai-ml/2026-10-15-backpropagation/ in the site's repository.

Cite this page

@misc{sigmoid-backpropagation,
  title  = {Anatomy of Backpropagation},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/backpropagation}
}