Dissection · He et al., 2016

Anatomy of the ResNet

Deep Residual Learning, dissected: the paper's own 20-layer CIFAR network in 3D, every feature map and kernel a real one from a model trained on handwritten digits and running in your browser, every number from the paper traced to its sentence.

PaperHe et al., CVPR 2016

Model67,906 parameters, live

Reading26 min · 27 steps

Published2026-10-17

01Overview

A network that is allowed to do nothing

In December 2015 four researchers at Microsoft Research reported that a neural network could be made much deeper, and much better, if every pair of layers was given a way around itself. They trained image classifiers up to 152 layers deep, won the ImageNet classification challenge with 3.57% top-5 error, and trained a 1,202-layer network that showed no optimisation difficulty [1]. The bypass, the identity shortcut, is now in almost every deep network, including every Transformer [2].

What you are looking at is one of the paper's own networks, whole: the 20-layer residual network of its CIFAR-10 experiments, at half width, trained on handwritten digits. The digit enters at the bottom. The stream of feature maps rises up the left-hand spine, block by block; each block's two convolutions are built out to the right, with their kernels laid out below the maps they produce; the violet rails beside the spine are the shortcuts. At the top, a classifier turns the last maps into ten probabilities.

It is not a diagram. Every block is a real tensor from a trained network running in this page, and every cell is one number: its colour and height are the value. Hover any cell to read it and see where it came from. The page's input is .

How to use this page

Scroll the article, or use ← →. Drag the stage to orbit, right-drag to pan, scroll on it to zoom; O shows the whole machine, C returns to the step's view, R replays its animation. Change the digit number in the bar above and every map is recomputed.

Go deeper: Residual connections · Convolutional neural networks

02Overview

The paper's CIFAR network, at half width, on 8×8 digits

The paper's ImageNet networks are too large to run in a browser, but its CIFAR-10 family is small by design: the authors "intentionally use simple architectures" there, to study depth rather than to set records. A first 3×3 convolution, then three stages of $2 n$ layers on maps of sizes 32, 16 and 8 with 16, 32 and 64 filters, then global average pooling, a 10-way fully connected layer and softmax: 6n+2 weighted layers in all [1].

This page keeps that design with $n = 3$ , the paper's ResNet-20, and halves every width: 8, 16 and 32 filters on 8×8, 4×4 and 2×2 maps. It is trained on the UCI handwritten digits [3], 1,797 images of 8×8 pixels with 17 grey levels, made by counting the inked pixels in 4×4 blocks of a 32×32 bitmap. They come from 43 writers and are split per class into 1,438 training and 359 held-out digits, so every digit is in both halves; CIFAR-10 has 50,000 training and 10,000 test images. The recipe is the paper's, with sizes changed: the details are in the training step.

The network has parameters and gets of the held-out digits wrong. Its plain twin, the same network without shortcuts, has exactly as many parameters and gets wrong. At 20 layers the two are close. The point of the paper, and of this page, is what happens as they get deeper.

03The problem

Why would a deeper network fit its training data worse?

By 2015, depth was known to matter, and the old obstacle to it, gradients that vanish or explode, had been largely addressed by careful initialisation [4] and batch normalisation [5] [1]. Networks of a few dozen layers could start to train. What happened next surprised the authors: "with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly."

The surprising part is where it degrades. "Such degradation is not caused by overfitting": a deeper network is worse on the data it was trained on, not only on new data [1]. Overfitting would be the reverse.

And it should not happen, because a deeper network can always do at least as well as a shallower one. "There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model." The figure below builds that solution from this page's plain 20-layer net: add identity layers and nothing changes. Then it sets beside it a 56-layer plain net trained from scratch with the same recipe on the same digits.

The deeper net could have matched the shallower one and did not. The fault lies in the optimiser's ability to find the solution, not in what the network can express. On the stage the shortcut rails fade out: what remains is a plain network, the kind that degrades.

Go deeper: Signal propagation and initialisation · Optimisation theory

04The problem

Learn the change, not the whole mapping

Call the mapping a few layers should compute $H (x)$ . Instead of asking them to compute it, the paper asks them for the difference: "we let the stacked nonlinear layers fit another mapping of F(x) := H(x) − x". The input is then added back. The network still computes $H (x) = F (x) + x$ ; what the layers learn is the residual.

y = F (x, {W_{i}}) + x (1)

Here $x$ and $y$ are the input and output of a block and $F$ is what its layers compute: for the two-layer blocks on this page, $F = W_{2} σ (W_{1} x)$ with $σ$ the ReLU, batch normalisation included and biases omitted [1]. Equation (1) adds the input to the output, channel by channel; on the stage that addition happens where the orange link from $F (x)$ meets the spine.

Why should that be easier? If the best thing a block can do is nothing, "it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers." In practice identity is rarely optimal, but if the needed function is close to identity, small corrections to it should be easier to find than the whole function: the reformulation "may help to precondition the problem" [1]. The paper never proves this; footnote 2 calls the assumption that layers can approximate such functions "still an open question".

The cost of the shortcut is nothing. "Identity shortcut connections add neither extra parameter nor computational complexity", which is what makes the paper's experiments clean: a plain network and its residual twin have the same number of parameters, depth, width and computational cost, and differ only in the addition.

Go deeper: Residual connections · The residual stream

05The computation

What the network actually receives

The input is . On the left of the stage it is drawn as it is stored, 64 bars whose height is the ink in each 4×4 block of the original scan, scaled to 0–1.

The network does not see that image. As in the paper, whose CIFAR inputs are 32×32 "with the per-pixel mean subtracted", every pixel has the average of that pixel over the 1,438 training digits taken away. The block to its right is the result: amber where this digit has more ink than usual, blue where it has less. The digits' borders are almost always empty, so the corners are near zero; the strokes in the middle carry the signal.

On CIFAR this is the only preprocessing the paper describes. Batch normalisation, inside the network, does the rest.

06The computation

Eight filters slide over the digit

The first layer is a 3×3 convolution (§4.2). Each of its 8 filters is a 3×3 grid of weights, drawn as one 3-wide column of the kernel block on the stage. A filter is laid on every 3×3 patch of the input in turn, multiplied cell by cell and summed, giving one number per position: an 8×8 map per filter, eight maps in all. Zero padding around the border keeps the map 8×8.

z_{c} (i, j) = u = - 1 \sum 1 v = - 1 \sum 1 W_{c} (u, v) x (i + u, j + v)

The animation lights one patch and one filter and reveals the single value they make, then fills in the rest. The eight maps are then batch-normalised and passed through a ReLU, $max (0, z)$ ; the result is the start of the residual stream, the first block on the spine.

Every convolution in the network is followed by batch normalisation, as the paper specifies: "right after each convolution and before activation" [1]. That is why the convolutions have no bias: BN's own shift replaces it.

Go deeper: Convolutional neural networks · Activation functions

07The computation

Inside a block: every channel, one filter

The first block reads the 8 maps on the spine. Its first convolution has 8 filters again, but each now spans all 8 input channels: 8 × 3×3 = 72 weights per filter. On the stage the kernel block is 24 rows deep, one 3-row band per input channel, and each 3-wide column is one filter. One output value is the sum of 72 products: the 3×3 patch at that position in every input map, times the filter.

The figure computes one value by hand from your digit and checks it against the engine: pick a filter and a pixel. The sum, then batch norm, then the ReLU give the value drawn at that cell of the map above the kernel. The paper's networks are made of nothing but this operation, repeated: 3×3 filters throughout, following two rules borrowed from VGG: "(i) for the same output feature map size, the layers have the same number of filters; and (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer" [1].

08The computation

Batch normalisation: four numbers per channel

Each convolution's output goes through batch normalisation [5]. During training, every channel is standardised over the mini-batch, then rescaled by a learned $γ$ and shifted by a learned $β$ . At test time, the batch statistics are replaced by running averages kept during training, so the operation is a fixed affine map per channel:

BN (z) = γ \frac{z - μ}{σ ^{2} + ϵ} + β

The small four-row block under each map on the stage holds exactly these numbers for its channels: $γ$ , $β$ , the running mean $μ$ and the running standard deviation $σ$ . The animation steps through the rows, then reveals the normalised, rectified map channel by channel.

The paper's argument depends on BN. The plain networks' difficulty, the authors say, is "unlikely to be caused by vanishing gradients", because with BN gradient norms stay healthy, which they checked [1]. Notice also $γ$ : it sets the scale of everything the branch adds to the stream. It returns in the research desk.

Go deeper: Batch norm and layer norm

09The computation

The second convolution makes F(x)

The second convolution reads the rectified maps of the first and is batch-normalised again, but this time there is no ReLU after it. Its output is $F (x)$ , the residual, drawn in the right-hand column: the change this block proposes to the stream. It can be negative as well as positive, which is why it is drawn in blue and amber both.

The paper tried a residual function with a single layer and found that "if F has only a single layer, Eqn.(1) is similar to a linear layer: y = W1 x + x, for which we have not observed advantages". Two layers, with a nonlinearity between them, is the smallest residual function that helps. ImageNet's deeper networks use three [1].

10The computation

Add the input back, then rectify

Now the shortcut. The violet rail carries the block's input around both convolutions, unchanged, and the two are added element by element: $F (x) + x$ . Then the second ReLU: "We adopt the second nonlinearity after the addition (i.e., σ(y), see Fig. 2)" [1]. The block's output is

y = σ (F (x) + x) .

Hover a cell of the block's output to see its three numbers: the residual, the shortcut and the rectified sum. Where the residual is near zero, the output equals the input. Where it is large and negative, the ReLU cuts the sum to zero, and that is the one place information on the shortcut can be lost.

11The computation

Three blocks on 8×8 maps

Stage 1 repeats the block three times on 8×8 maps with 8 channels. Each block reads the spine, proposes a residual and adds it. The spine is the running sum: after three blocks it holds the stem's output plus three corrections, each passed through a ReLU.

The paper connects shortcuts "to the pairs of 3×3 layers (totally 3n shortcuts)" and uses identity shortcuts everywhere on CIFAR (option A, below). With $n = 3$ there are 9 shortcuts and 18 convolutional layers in the stages; with the first convolution and the classifier, 20 weighted layers.

In 2017 the Transformer took over the same pattern with layer normalisation in place of BN: each of its sub-layers has a residual connection around it, followed by layer normalisation, citing this paper for the connection [2]. The spine on this stage is what interpretability researchers call the residual stream.

Go deeper: LayerNorm and residual connections · The residual stream · Transformer architecture

12The computation

Halve the map, double the channels: what happens to the shortcut?

Block 4 starts stage 2. Its first convolution moves in steps of two pixels, "with a stride of 2", so its output map is 4×4, and it has 16 filters, following the rule that halving the map doubles the filters. The animation shows one output pixel reading a patch from twice as far across the input.

Now $F (x)$ is 4×4×16 and $x$ is 8×8×8: equation (1) cannot add them. The paper offers two fixes. "(A) The shortcut still performs identity mapping, with extra zero entries padded for increasing dimensions. This option introduces no extra parameter"; or "(B) The projection shortcut in Eqn.(2) is used to match dimensions (done by 1×1 convolutions)", adding a learned matrix $W_{s}$ :

y = F (x, {W_{i}}) + W_{s} x (2)

On CIFAR the paper uses option A throughout, and so does this page. The orange rail takes every second pixel of every second row (stride 2), and the shortcut map on the spine, between the block's input and its output, shows the result: the first 8 channels are the input, subsampled; the other 8 are zeros. Hover them.

13The computation

Stage 2: smaller maps, more channels, same cost

Blocks 4 to 6 work on 4×4 maps with 16 channels. Each map has a quarter of the positions and each filter twice the inputs and there are twice the filters, so a layer costs the same number of multiply-adds as in stage 1: $4 \times 4 \times 16 \times 16 \times 9 = 8 \times 8 \times 8 \times 8 \times 9 = 36, 864$ . That is design rule (ii) at work, keeping the time complexity per layer constant.

Memory is not constant: each stage's maps hold half the numbers of the stage before, which is why the spine blocks on the stage get thinner as they rise.

14The computation

Stage 3: where the parameters are

Blocks 7 to 9 run on 2×2 maps with 32 channels, again through an option A shortcut at block 7. The kernel blocks here are the largest on the table: a 32-to-32 filter bank is $32 \times 32 \times 9 = 9, 216$ weights, four times stage 2's and sixteen times stage 1's. Constant computation per layer means parameters grow fourfold per stage. The three blocks of stage 3 hold 75% of the network's weights while touching maps with only 4 positions each.

The same pattern sets the paper's sizes: 0.27M parameters for its ResNet-20 with 16, 32 and 64 filters, four times this page's 67,906 at twice the width.

Go deeper: Depth versus width

15The computation

Average each map to one number

After the last block, each of the 32 channels is a 2×2 map. Global average pooling replaces each map with its mean: 32 numbers, one per channel, whatever the size of the image. There are no weights here, and none of the large fully connected layers older networks used at this point; the paper's ImageNet networks end the same way.

16The computation

Ten scores, then ten probabilities

A 10-way fully connected layer multiplies the 32 pooled numbers by a 32×10 weight matrix and adds a bias, giving one score, a logit, per digit. The matrix sits on the stage directly under the logits it produces, column $j$ under the score for digit $j$ . Softmax turns the scores into probabilities that sum to 1:

p_{k} = \frac{e ^{z_{k}}}{\sum _{j = 0}^{9} e ^{z_{j}}}

Try digit #14 or #346, two that the network gets wrong.

Go deeper: Softmax and logits

17Design choices

Is the identity shortcut essential, or would any shortcut do?

On ImageNet the paper compares three shortcut designs at 34 layers: A, zero padding wherever dimensions grow; B, 1×1 projections only there; C, projections on every shortcut. The residual nets beat the plain one in every case, and the three differ by less than a point: 25.03%, 24.52% and 24.19% top-1.

"B is slightly better than A. We argue that this is because the zero-padded dimensions in A indeed have no residual learning." "C is marginally better than B", which the authors attribute to the extra parameters of its thirteen projections. Their conclusion: "projection shortcuts are not essential for addressing the degradation problem" [1]. What matters is that a path exists that the layers do not have to learn.

The two option A blocks are highlighted on the stage. Under option A, half of a downsampling block's output channels start from zero on the shortcut side and must be built entirely by the residual: the paper's explanation for B's edge, and, as the research desk finds, a reason those two blocks behave differently from the others.

18Design choices

How can 152 layers cost less than 19?

The paper's ImageNet networks follow the same rules as the CIFAR ones at larger scale. Its 34-layer network "has 3.6 billion FLOPs (multiply-adds), which is only 18% of VGG-19 (19.6 billion FLOPs)" [1]. The figure recomputes those counts from the paper's Table 1, layer by layer.

Beyond 34 layers the block changes. A "bottleneck" block is a 1×1 convolution that cuts the channels by four, a 3×3 convolution on the narrow maps, and a 1×1 convolution that restores them. Three layers cost about what two 3×3 layers did. With it, "the 152-layer ResNet (11.3 billion FLOPs) still has lower complexity than VGG-16/19 nets (15.3/19.6 billion FLOPs)". The page's miniature uses the two-layer block, as the paper's CIFAR networks do. On this stage every weight block is highlighted: these, and the BN rows, are all the parameters there are.

19Training

How the paper trains it, and how this page did

On CIFAR-10 the paper trains "with a minibatch size of 128 on two GPUs", with stochastic gradient descent: "We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and terminate training at 64k iterations". Weight decay is 0.0001, momentum 0.9, weights start from the He initialisation, there is BN and no dropout. The only augmentation is "4 pixels are padded on each side, and a 32×32 crop is randomly sampled from the padded image or its horizontal flip" [1].

The He initialisation draws each weight from a zero-mean Gaussian whose standard deviation is $2/ n_{l}$ , with $n_{l} = k^{2} c$ the number of inputs to a unit in He et al.'s derivation [4]. It keeps the variance of a ReLU network's activations roughly constant from layer to layer at the start of training.

This page's nets used the same optimiser, momentum, weight decay, batch size, initial learning rate and initialisation. 64,000 iterations of 128 images over 50,000 is 164 epochs; 164 epochs of 1,438 digits is 1,840 iterations, so training stops there, with the two learning-rate drops at the same fractions, 50% and 75%. Augmentation pads by 1 pixel of 8 and crops; there is no flip. Each 20-layer net trained in under a minute on one CPU thread.

Go deeper: Learning-rate schedules · Weight decay · Dropout · SGD

20Why it works

Why do shortcuts make deep networks trainable?

The paper's own explanation is preconditioning, and it is careful not to claim more. It argues the plain nets' difficulty is not vanishing gradients and conjectures instead that "the deep plain nets may have exponentially low convergence rates"; training them three times longer still showed degradation [1].

The authors' follow-up gave the clearest picture [7]. If every block were exactly $x_{l + 1} = x_{l} + F (x_{l})$ , then for any deeper block $L$ and shallower block $l$ ,

x_{L} = x_{l} + i = l \sum L - 1 F (x_{i}, W_{i})

\frac{\partial E}{\partial x _{l}} = \frac{\partial E}{\partial x _{L}} (1 + \frac{\partial}{\partial x _{l}} i = l \sum L - 1 F (x_{i}, W_{i}))

The deep feature is the shallow one plus a sum of corrections, and the gradient at any block contains the gradient at the top unchanged: the "1" in the bracket. In a plain network the same quantity is a product of one Jacobian per layer. The rails on the stage draw this clean path. In this paper's block it is not quite clean: the function after the addition is a ReLU, which cuts every negative sum to zero.

A third account looks at the gradient's shape rather than its size. Balduzzi and colleagues showed that "as depth increases, gradients in standard feedforward networks increasingly resemble white noise", while residual networks' stay correlated across nearby inputs [8]. The toy above, random ReLU networks with a normalisation step standing in for BN, shows a mild version of the effect: drag the depth and the plain gradient roughens while the residual one stays smooth.

Normalisation is not the only way to keep a deep residual net trainable. Fixup rescales the initialisation of the residual branches instead, and trains residual networks without normalisation as stably as with it, "even for networks with 10,000 layers" [10].

Go deeper: Backpropagation · Calculus and gradients · Loss landscapes

21Why it works

One deep network, or many shallow ones?

Unroll equation (1) and a residual network of $n$ blocks is a sum over every way of entering or skipping each block: "residual networks have 2^n paths connecting input to output layers", of every length from 0 to $n$ [9]. The stage lights one such path at a time: the blocks whose branch it enters. The page's network has 9 blocks and 512 paths.

The path lengths follow a binomial distribution: in a 54-block ResNet-110, "more than 95% of paths go through 19 to 35 modules". And the gradient during training comes mostly from shorter ones, "paths that are only 10-34 layers deep". Veit and colleagues tested the consequence directly. Delete one layer from a plain VGG network at test time and "deleting any layer in VGG reduces performance to chance levels"; delete one block from a ResNet and performance is barely affected. That claim, on this page's architecture, is the research desk's second experiment.

Stochastic depth [11] made the idea a training method: "randomly drop a subset of layers and bypass them with the identity function" for each mini-batch. Wide residual networks pushed the other way, showing that "a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks" [12].

22Results

What the shortcuts bought

On ImageNet, the 34-layer plain network has a higher validation error than the 18-layer one: 27.94% against 28.54%, top-1. With shortcuts, the order reverses: "the 34-layer ResNet is better than the 18-layer ResNet (by 2.8%)", and it beats its own plain twin by 3.5 points. At 18 layers the plain and residual nets are equally accurate; the ResNet converges faster [1]. Deeper bottleneck nets improved further: ResNet-152 reached 4.49% single-model top-5 validation error, and an ensemble 3.57% on the test set.

On CIFAR-10, "the deep plain nets suffer from increased depth, and exhibit higher training error when going deeper", and plain-110 is above 60% error. The residual nets improve with depth up to 110 layers, which needed one change: "we use 0.01 to warm up the training until the training error is below 80% (about 400 iterations), and then go back to 0.1".

At 1,202 layers training is still easy, training error below 0.1%, but test error rises to 7.93%. The authors attribute it to size: "We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset." The representations transferred too: replacing VGG-16 with ResNet-101 in a detector gave a 28% relative improvement on COCO.

23Results

Counting the weights by hand

Every parameter of the paper's CIFAR networks is in a 3×3 convolution, a BN layer or the classifier: option A shortcuts add none. For $n$ blocks per stage, 3 input channels and widths 16, 32, 64:

P (n) = first conv 9 \cdot 3 \cdot 16 + stages \sum (9 c_{in} c + (2 n - 1) 9 c^{2}) + BN γ, β 2 \cdot (16 + 2 n (16 + 32 + 64)) + classifier 64 \cdot 10 + 10 .

The tests recompute it: 269,722 for ResNet-20, 464,154 for ResNet-32, and so on up to 19.4 million for ResNet-1202, matching every entry of Table 6 to the precision printed: 0.27M, 0.46M, 0.66M, 0.85M, 1.7M and 19.4M. With 1 input channel and widths 8, 16, 32 the same formula gives this page's . On the stage the weight blocks are highlighted: stage 3's six filter banks are three quarters of them.

24Research desk

Does the degradation problem appear on a dataset this small?

The paper shows degradation on CIFAR-10, with 50,000 images, and ImageNet, with a million. On 1,438 digits, every network here can in principle memorise its training set, so it is a fair question whether the optimisation difficulty appears at all. I wrote the hypotheses and verdict rules in research/README.md before running anything, trained plain and residual nets at the paper's four depths, 20, 32, 44 and 56 layers, with three seeds each (24 networks, the recipe above, 1,840 iterations), and scored the final training error on all 1,438 training digits.

Claim

"The deep plain nets suffer from increased depth, and exhibit higher training error when going deeper", while ResNets "manage to overcome the optimization difficulty" (§4.2).

Hypotheses

H1: plain-56 ends at least 1 point worse than plain-20 in training error. H2: ResNet-56 ends no more than 0.2 points worse than ResNet-20. H3: ResNet-56 ends at least 1 point better than plain-56.

Design

n = 3, 5, 7, 9; plain and residual; seeds 0, 1, 2; training error in eval mode on un-augmented training digits. Partial-support thresholds are in the README.

25Research desk

Can you delete a block from a trained ResNet?

Claim

Removing one block of a trained ResNet at test time "does not noticeably affect their performance", while removing a layer of a plain network drops it to chance; downsampling blocks have a modest impact; and error "increases smoothly" as more blocks go [9].

Why test it again

Veit et al. lesioned pre-activation ResNets and a 15-layer VGG with 128 channels everywhere. This page's net is the original post-activation block of He et al. (2016), and its plain twin has identical layers. In the original block, deleting

F

leaves

y = σ (x) = x

exactly, because

x

is itself the output of a ReLU; the tests check this.

Hypotheses

H4: ≥ 90% of single ordinary-block deletions cost ResNet-56 under 5 points of held-out error, and ≥ 90% of single deletions leave plain-56 above 50%. H5: deleting a downsampling block costs ≥ 2× an ordinary one. H6: deleting k random blocks, k = 1 to 12, raises error monotonically (Spearman ρ ≥ 0.9) with no step over 25 points.

The live panel runs the same lesion on the page's 20-layer nets, and the shorter net is less forgiving. Deleting block 1 of ResNet-20 takes its held-out error from 1.7% to 82.5%, and block 2 to 45.1%; deleting block 8 changes nothing at all. Every single deletion from plain-20 leaves it between 67.7% and 91.4%. In ResNet-56 the same early blocks were also the costliest: E2's worst deletion was block 1. The stage dims one block's branch at a time.

26Research desk

Are the residual functions closer to zero?

Claim

"The responses are the outputs of each 3×3 layer, after BN and before other nonlinearity (ReLU/addition)." "ResNets have generally smaller responses than their plain counterparts", and "an individual layer of ResNets tends to modify the signal less" as depth grows (§4.2, Fig. 7). The paper offers this as support for its preconditioning argument.

Hypotheses

H7: the median response std of each ResNet is below 0.8× its plain twin's at both 20 and 56 layers, in every seed. H8: ResNet-56's median is below 0.9× ResNet-20's.

Design

For the E1 nets at 20 and 56 layers (3 seeds × 2 arms × 2 depths), the standard deviation of every 3×3 layer's response over the 359 held-out digits, in eval mode.

On the stage, the $γ$ row of every block's second batch norm is lit: the learned scale on the last layer of each residual branch, and so on everything the branch adds to the stream.

27Research desk

What this suggests, and what to test next

Each of these can be run with the scripts in this page's research/ directory:

Where does the cliff come from? Plain nets here are fine at 32 layers and broken at 44. Train plain-38 and plain-44 for 3× and 10× the iterations: if they recover, the paper's "exponentially low convergence rates" conjecture fits; if not, the plain nets are stuck, not slow. The paper's own 3× test is one point on that curve.
Pre-activation and the lesion study. Repeat E2 with the pre-activation block of He et al. (2016b). With no ReLU after the addition, are single deletions cheaper still?
Does option B fix the downsampling blocks? Train with 1×1 projections at the two downsampling blocks and re-run E2's H5. If the extra cost of deleting them disappears, it was the zero-padded channels, as the paper's explanation of B beating A would predict.
Is Fig. 7 about γ? Freeze every BN $γ$ at 1 and repeat E3. If residual and plain responses become equal, the effect the paper reports lives in BN's scale, which Fixup sets by hand: it begins by asking to "initialize the classification layer and the last layer of each residual branch to 0" [10].
Ensemble or not? Veit et al.'s strongest evidence is smooth multi-block deletion. Compare it against a true ensemble of shallow nets of matched size on the same digits: does the ResNet's error-versus-k curve have the same shape?

The miniature, its weights and every experiment are in the page's directory; the same JavaScript that draws the stage is what the tests check against PyTorch. Corrections and replications are welcome.

Go deeper: Vision Transformers · Model selection

References

He, K., Zhang, X., Ren, S. and Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR 2016; arXiv:1512.03385v1 The primary source. Quotes are from the text of the arXiv v1 PDF (10 Dec 2015), fetched from the arXiv bulk-data mirror on Google Cloud Storage because arxiv.org was unreachable from the build environment; no LaTeX source was used.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. (2017). Attention Is All You Need. NIPS 2017; arXiv:1706.03762v7
Alpaydin, E. and Kaynak, C. (1998). Optical Recognition of Handwritten Digits (UCI Machine Learning Repository). UCI Machine Learning Repository; the copy bundled with scikit-learn 1.9.1 as load_digits Quotes are from the dataset description shipped with scikit-learn.
He, K., Zhang, X., Ren, S. and Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV 2015; arXiv:1502.01852v1 The initialisation the ResNet paper adopts ([13] there).
Ioffe, S. and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML 2015; arXiv:1502.03167v3
Srivastava, R. K., Greff, K. and Schmidhuber, J. (2015). Highway Networks. ICML 2015 Deep Learning Workshop; arXiv:1505.00387v2
He, K., Zhang, X., Ren, S. and Sun, J. (2016). Identity Mappings in Deep Residual Networks. ECCV 2016; arXiv:1603.05027v3 Quotes from the v3 PDF text.
Balduzzi, D., Frean, M., Leary, L., Lewis, J. P., Ma, K. W.-D. and McWilliams, B. (2017). The Shattered Gradients Problem: If resnets are the answer, then what is the question?. ICML 2017; arXiv:1702.08591v2
Veit, A., Wilber, M. and Belongie, S. (2016). Residual Networks Behave Like Ensembles of Relatively Shallow Networks. NIPS 2016; arXiv:1605.06431v2 Quotes from the v2 PDF text.
Zhang, H., Dauphin, Y. N. and Ma, T. (2019). Fixup Initialization: Residual Learning Without Normalization. ICLR 2019; arXiv:1901.09321v2
Huang, G., Sun, Y., Liu, Z., Sedra, D. and Weinberger, K. Q. (2016). Deep Networks with Stochastic Depth. ECCV 2016; arXiv:1603.09382v3
Zagoruyko, S. and Komodakis, N. (2016). Wide Residual Networks. BMVC 2016; arXiv:1605.07146v4

Fact ledger

Every number from the literature on this page, with the sentence it comes from. Hover any underlined number in the text to see its entry.

Value	What	Source
152 layers	deepest ImageNet ResNet	[1] Abstract On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [41] but still having lower complexity.
1st place, classification	ILSVRC 2015	[1] Abstract This result won the 1st place on the ILSVRC 2015 classification task.
3.57%	ILSVRC 2015 ensemble, top-5 test error	[1] Abstract; Table 5 An ensemble of these residual nets achieves 3.57% error on the ImageNet test set.
no optimisation difficulty	ResNet-1202	[1] §4.2 Our method shows no optimization difficulty
to study depth	why the CIFAR nets are simple	[1] §4.2 Our focus is on the behaviors of extremely deep networks, but not on pushing the state-of-the-art results, so we intentionally use simple architectures as follows.
6n layers on {32, 16, 8} maps	the CIFAR stack	[1] §4.2 Then we use a stack of 6n layers with 3×3 convolutions on the feature maps of sizes {32, 16, 8} respectively, with 2n layers for each feature map size.
{16, 32, 64}	CIFAR filters	[1] §4.2 The numbers of filters are {16, 32, 64} respectively.
global average pooling, 10-way fc, softmax	the CIFAR head	[1] §4.2 The network ends with a global average pooling, a 10-way fully-connected layer, and softmax.
6n+2	weighted layers	[1] §4.2 There are totally 6n+2 stacked weighted layers.
1,797	how many	[3] dataset description :Number of Instances: 1797
1,797 images, 8×8, 0..16	the digits	[3] dataset description 8x8 image of integer pixels in the range 0..16.
4×4 blocks of a 32×32 bitmap	how the 8×8 images were made	[3] dataset description 32x32 bitmaps are divided into nonoverlapping blocks of 4x4 and the number of on pixels are counted in each block.
43 people	writers	[3] dataset description From a total of 43 people, 30 contributed to the training set and different 13 to the test set.
50k train, 10k test, 10 classes	CIFAR-10	[1] §4.2 which consists of 50k training images and 10k testing images in 10 classes.
largely addressed	the old obstacle	[1] §1 This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16]
degrades rapidly	the degradation problem	[1] §1 with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly.
higher training error	not overfitting	[1] §1 Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error
identity layers	the solution by construction	[1] §1 There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model.
F(x) := H(x) − x	the residual mapping	[1] §1 we let the stacked nonlinear layers fit another mapping of F(x) := H(x) − x.
push the residual to zero	why residuals are easier	[1] §1 To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.
identity as reference	preconditioning	[1] §3.1 In real cases, it is unlikely that identity mappings are optimal, but our reformulation may help to precondition the problem.
open question	the approximation hypothesis	[1] §3.1, footnote 2 This hypothesis, however, is still an open question.
none	cost of an identity shortcut	[1] §1 Identity shortcut connections add neither extra parameter nor computational complexity.
same parameters, depth, width, cost	a fair comparison	[1] §3.2 We can fairly compare plain/residual networks that simultaneously have the same number of parameters, depth, width, and computational cost (except for the negligible element-wise addition).
transform and carry	highway gates	[6] §2 We refer to T as the transform gate and C as the carry gate
data-dependent, with parameters	highway gates	[1] §2 These gates are data-dependent and have parameters, in contrast to our identity shortcuts that are parameter-free.
32×32, per-pixel mean subtracted	CIFAR input	[1] §4.2 The network inputs are 32×32 images, with the per-pixel mean subtracted.
3×3 convolutions	the first layer	[1] §4.2 The first layer is 3×3 convolutions.
after each convolution, before activation	where BN goes	[1] §3.4 We adopt batch normalization (BN) [16] right after each convolution and before activation, following [16].
same filters per size; double filters when halved	the two design rules	[1] §3.3 (i) for the same output feature map size, the layers have the same number of filters; and (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer.
unlikely vanishing gradients	the cause, per the authors	[1] §4.1 We argue that this optimization difficulty is unlikely to be caused by vanishing gradients.
healthy	gradient norms with BN	[1] §4.1 We also verify that the backward propagated gradients exhibit healthy norms with BN.
no advantage observed	a one-layer residual function	[1] §3.2 But if F has only a single layer, Eqn.(1) is similar to a linear layer: y = W1 x + x, for which we have not observed advantages.
after the addition	where the second ReLU goes	[1] §3.2 We adopt the second nonlinearity after the addition (i.e., σ(y), see Fig. 2).
7.61% vs 4.92%	ResNet-1001, original vs proposed unit	[7] Fig. 1 ResNet−1001, original (error: 7.61%) ResNet−1001, proposed (error: 4.92%)
4.62% on CIFAR-10	ResNet-1001 with pre-activation	[7] Abstract We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100
eases optimisation	the clean path	[7] §1 These experiments suggest that keeping a “clean” information path (indicated by the grey arrows in Fig. 1, 2, and 4) is helpful for easing optimization.
3n, all option A	CIFAR shortcuts	[1] §4.2 When shortcut connections are used, they are connected to the pairs of 3×3 layers (totally 3n shortcuts). On this dataset we use identity shortcuts in all cases (i.e., option A),
around each sub-layer	residuals in the Transformer	[2] §3.1 We employ a residual connection [11] around each of the two sub-layers, followed by layer normalization [1].
stride 2	shortcuts across map sizes	[1] §3.3 For both options, when the shortcuts go across feature maps of two sizes, they are performed with a stride of 2.
identity with zero padding	option A shortcut	[1] §3.3 (A) The shortcut still performs identity mapping, with extra zero entries padded for increasing dimensions. This option introduces no extra parameter;
1×1 projection	option B shortcut	[1] §3.3 (B) The projection shortcut in Eqn.(2) is used to match dimensions (done by 1×1 convolutions).
0.27M params, 8.75%	ResNet-20 on CIFAR-10	[1] Table 6 ResNet 20 0.27M 8.75
25.03% top-1, 7.76% top-5	ResNet-34 option A	[1] Table 3 ResNet-34 A 25.03 7.76
24.52% top-1, 7.46% top-5	ResNet-34 option B	[1] Table 3 ResNet-34 B 24.52 7.46
24.19% top-1, 7.40% top-5	ResNet-34 option C	[1] Table 3 ResNet-34 C 24.19 7.40
zero-padded dimensions	why B beats A	[1] §4.1 B is slightly better than A. We argue that this is because the zero-padded dimensions in A indeed have no residual learning.
thirteen	projection shortcuts in C	[1] §4.1 C is marginally better than B, and we attribute this to the extra parameters introduced by many (thirteen) projection shortcuts.
not essential	projections	[1] §4.1 But the small differences among A/B/C indicate that projection shortcuts are not essential for addressing the degradation problem.
“expect that”	a typo in the source	[1] §4.1 The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3×3 filters as in Fig. 3 (right).
3.6 billion FLOPs, 18% of VGG-19 (19.6 billion)	34-layer baseline cost	[1] §3.3 Our 34-layer baseline has 3.6 billion FLOPs (multiply-adds), which is only 18% of VGG-19 (19.6 billion FLOPs).
1×1, 3×3, 1×1	bottleneck block	[1] §4.1 The three layers are 1×1, 3×3, and 1×1 convolutions, where the 1×1 layers are responsible for reducing and then increasing (restoring) dimensions
11.3 billion FLOPs	ResNet-152 cost	[1] §4.1 the 152-layer ResNet (11.3 billion FLOPs) still has lower complexity than VGG-16/19 nets (15.3/19.6 billion FLOPs).
128 on two GPUs	CIFAR mini-batch	[1] §4.2 These models are trained with a minibatch size of 128 on two GPUs.
0.1, ÷10 at 32k and 48k, stop at 64k	CIFAR schedule	[1] §4.2 We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and terminate training at 64k iterations, which is determined on a 45k/5k train/val split.
weight decay 0.0001, momentum 0.9, no dropout	CIFAR regularisation	[1] §4.2 We use a weight decay of 0.0001 and momentum of 0.9, and adopt the weight initialization in [13] and BN [16] but with no dropout.
4-pixel pad, random crop, flip	CIFAR augmentation	[1] §4.2 4 pixels are padded on each side, and a 32×32 crop is randomly sampled from the padded image or its horizontal flip.
std √(2/n_l)	He initialisation	[4] §2.2 Gaussian distribution whose standard deviation (std) is 2/nl . This is our way of initialization.
256	ImageNet mini-batch	[1] §3.4 We use SGD with a mini-batch size of 256.
0.0001 and 0.9	weight decay and momentum	[1] §3.4 We use a weight decay of 0.0001 and a momentum of 0.9.
none	dropout	[1] §3.4 We do not use dropout [14], following the practice in [16].
exponentially low convergence rates	the authors' conjecture	[1] §4.1 We conjecture that the deep plain nets may have exponentially low convergence rates
3× iterations, still degraded	longer training	[1] §4.1, footnote 3 We have experimented with more training iterations (3×) and still observed the degradation problem
ReLU after the addition	the original unit's f	[7] §1 In [1], h(xl) = xl is an identity mapping and f is a ReLU [2] function.
white noise in plain nets	shattered gradients	[8] §1 The shattering gradients problem is that, as depth increases, gradients in standard feedforward networks increasingly resemble white noise.
short paths avoid vanishing gradients	Veit et al.'s conclusion	[9] Abstract Residual networks avoid the vanishing gradient problem by introducing short paths which can carry gradient throughout the extent of very deep networks.
10,000 layers without normalisation	Fixup	[10] Abstract We find training residual networks with Fixup to be as stable as training with normalization — even for networks with 10,000 layers.
2^n	paths in an n-block ResNet	[9] §3 It follows that residual networks have 2^n paths connecting input to output layers.
> 95% go through 19 to 35 modules	path lengths in ResNet-110	[9] §5 Figure 6 (a) shows the path length distribution for a residual network with 54 modules; more than 95% of paths go through 19 to 35 modules.
paths 10 to 34 layers deep	where the gradient comes from	[9] Abstract For example, most of the gradient in a residual network with 110 layers comes from paths that are only 10-34 layers deep.
chance level	dropping one layer of VGG	[9] §4.1 As expected, deleting any layer in VGG reduces performance to chance levels.
no noticeable effect	dropping one layer of a ResNet	[9] §1 Third, in initial experiments, we observe that removing single layers from residual networks at test time does not noticeably affect their performance.
drop whole blocks in training	stochastic depth	[11] Abstract We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function.
16 layers beat 1000	wide ResNets	[12] Abstract we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks
27.94% / 27.88%	plain-18 / ResNet-18 top-1 error	[1] Table 2 18 layers 27.94 27.88
28.54% / 25.03%	plain-34 / ResNet-34 top-1 error	[1] Table 2 34 layers 28.54 25.03
2.8 points	ResNet-34 over ResNet-18	[1] §4.1 the 34-layer ResNet is better than the 18-layer ResNet (by 2.8%).
3.5 points	ResNet-34 over plain-34	[1] §4.1 ResNet reduces the top-1 error by 3.5% (Table 2)
faster convergence	18 layers: faster, not better	[1] §4.1 In this case, the ResNet eases the optimization by providing faster convergence at the early stage.
4.49%	ResNet-152 single model, top-5 val	[1] §4.1; Table 4 Our 152-layer ResNet has a single-model top-5 validation error of 4.49%.
higher training error with depth	plain nets on CIFAR	[1] §4.2 The deep plain nets suffer from increased depth, and exhibit higher training error when going deeper.
error above 60%	plain-110	[1] Fig. 6 caption The error of plain-110 is higher than 60% and not displayed.
0.01 until training error < 80%	ResNet-110 warm-up	[1] §4.2 So we use 0.01 to warm up the training until the training error is below 80% (about 400 iterations), and then go back to 0.1 and continue training.
< 0.1%	ResNet-1202 training error	[1] §4.2 Our method shows no optimization difficulty, and this 10^3-layer network is able to achieve training error <0.1% (Fig. 6, right).
19.4M params, 7.93%	ResNet-1202	[1] Table 6 ResNet 1202 19.4M 7.93
overfitting	why 1202 tests worse than 110	[1] §4.2 We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M) for this small dataset.
28%	COCO detection, relative gain	[1] Abstract Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset.
2.3M params, 7.54%	Highway network, 19 layers	[1] Table 6 Highway [42, 43] 19 2.3M 7.54 (7.72±0.16)
no demonstrated gains	highway nets at 100+ layers	[1] §2 have not demonstrated accuracy gains with extremely increased depth (e.g., over 100 layers).
0.46M params, 7.51%	ResNet-32	[1] Table 6 ResNet 32 0.46M 7.51
0.66M params, 7.17%	ResNet-44	[1] Table 6 ResNet 44 0.66M 7.17
0.85M params, 6.97%	ResNet-56	[1] Table 6 ResNet 56 0.85M 6.97
1.7M params, 6.43% best (6.61 ± 0.16 over 5 runs)	ResNet-110	[1] Table 6 ResNet 110 1.7M 6.43 (6.61±0.16)
20, 32, 44, 56	depths compared	[1] §4.2 We compare n = {3, 5, 7, 9}, leading to 20, 32, 44, and 56-layer networks.
accuracy gains with depth	ResNets on CIFAR	[1] §4.2 our ResNets manage to overcome the optimization difficulty and demonstrate accuracy gains when the depth increases.
a modest impact	downsampling blocks	[9] §4.1 Removing downsampling blocks does have a modest impact on performance
error rises smoothly	deleting many blocks	[9] §4.2 deleting increasing numbers of residual modules, increases error smoothly, Figure 5 (a).
pre-activation, 110 layers (54 modules)	the nets Veit et al. lesioned	[9] §4 For our CIFAR-10 experiments, we train a 110-layer (54-module) residual network with modules of the “pre-activation” type which contain batch normalization as first step.
15 layers, 128 channels	the VGG they lesioned	[9] §4.1 To compare to conventional convolutional neural networks, we train a VGG network with 15 layers, setting the number of channels to 128 for all layers to allow the removal of any layer.
3×3 layer output after BN, before ReLU/addition	what a response is	[1] §4.2 The responses are the outputs of each 3×3 layer, after BN and before other nonlinearity (ReLU/addition).
generally smaller	residual responses	[1] §4.2, Fig. 7 Fig. 7 shows that ResNets have generally smaller responses than their plain counterparts.
each layer modifies the signal less	deeper ResNets	[1] §4.2 When there are more layers, an individual layer of ResNets tends to modify the signal less.
zero-initialise each residual branch's last layer	Fixup's first rule	[10] §3 Initialize the classification layer and the last layer of each residual branch to 0.

How this was made

Everything on the stage is computed in the page from trained weights: a 67,906-parameter ResNet-20 and its plain twin with the paper's architecture (6n+2 layers, 3×3 convolutions, BN after every convolution, option A shortcuts, global average pooling) and recipe (SGD 0.1 with momentum 0.9 and weight decay 1e-4, batch 128, learning rate divided by 10 at 50% and 75% of 164 epochs, He initialisation, pad-and-crop augmentation), trained in PyTorch on a CPU. The JavaScript forward pass matches PyTorch to 3.5e-6 on reference digits, and its held-out error and every single-block lesion match PyTorch exactly (test.mjs, 80 checks); the same tests recompute the paper's Table 6 parameter counts, the 1.8 and 3.6 billion FLOPs of its ImageNet nets and VGG-19's 19.6 billion, Veit et al.'s path-length arithmetic and every number the text states by hand. Paper figures are quoted from the text of the arXiv v1 PDF and listed in the page's fact ledger; all 103 quotes were checked mechanically against the fetched sources. The research-desk experiments, their pre-registration and results are in research/.

Built with the dissection skill. The visual grammar of tensors as blocks of cells owes much to Brendan Bycroft's LLM Visualization. Source, weights, training scripts and experiments: anatomy/ai-ml/2026-10-17-resnet/ in the site's repository.

Cite this page

@misc{sigmoid-resnet,
  title  = {Anatomy of the ResNet},
  author = {Praveen T N},
  year   = {2026},
  note   = {Sigmoid Anatomy, interactive dissection},
  url    = {https://praveentn.live/anatomy/resnet}
}