Double Descent

For fifty years the standard picture of model capacity was a U. Too few parameters, high bias; too many, high variance; somewhere in between, an optimum you find by validation. Geman, Bienenstock, and Doursat wrote the canonical statement of this for neural networks in 1992, and every textbook since has reproduced the curve (Geman et al., 1992, Neural Networks and the Bias/Variance Dilemma, Neural Computation 4(1):1-58).

Modern practice ignores it entirely. Frontier models interpolate their training data and generalise anyway. Either the theory is wrong or practice is lucky, and it turns out the theory was incomplete.

The shape

Belkin et al. showed that the U is the left half of a longer curve (Belkin et al., 2019, Reconciling modern machine learning practice and the bias-variance trade-off, PNAS 116(32):15849-15854). Plot test error against model capacity and you get:

• Classical regime. Under-parameterised. Error falls, then rises, exactly as the textbook says.
• Interpolation threshold. Capacity just sufficient to fit the training data exactly. Test error peaks here, often above the error of much smaller models.
• Modern regime. Over-parameterised. Error falls again, frequently below the classical minimum, and keeps falling.

At the threshold there is essentially one interpolating solution and the fit is forced to be wild. Past it there are infinitely many, and the optimiser's implicit bias selects among them, typically toward small norm. More capacity means more freedom to interpolate smoothly, so error falls.

Nakkiran et al. established that this is not confined to linear models or synthetic data, and added two crucial generalisations (Nakkiran et al., 2020, Deep Double Descent, arXiv:1912.02292, ICLR 2020). Epoch-wise double descent: the same non-monotone shape appears as training time increases at fixed model size. Sample-wise non-monotonicity: for a fixed model, adding training data can hurt, if the additions move the model toward the interpolation threshold from the over-parameterised side.

They unify these with effective model complexity, defined as the largest number of samples the training procedure can fit to near-zero error. Double descent is monotone in the ratio of samples to effective model complexity, and the peak sits where that ratio is one.

Why interpolation is not fatal

The theory that explains the right half is benign overfitting. Bartlett et al. characterised exactly when a minimum-norm interpolating linear predictor generalises, in terms of two notions of effective rank of the data covariance (Bartlett et al., 2020, Benign overfitting in linear regression, PNAS 117(48):30063-30070).

The condition is a specific kind of high-dimensional structure: many low-variance directions, so that noise can be absorbed into directions that contribute little to prediction, while signal directions are fitted cleanly. Overfitting is benign when there is somewhere harmless to put the noise. When the spectrum decays too fast, there is nowhere to put it and interpolation is exactly as bad as classical theory predicted.

That condition is not automatic. It is a property of the data, and it explains why benign overfitting is common in high-dimensional natural data and rare in low-dimensional tabular problems.

When it breaks

The peak is real, and label noise makes it dramatic. On clean data double descent is often barely visible; Nakkiran et al. show the peak growing sharply with the fraction of corrupted labels. A practitioner who never sees the peak has probably never trained near the threshold on noisy data, not disproved the phenomenon.

Two practical consequences are worth internalising. First, a model that is slightly too big is dangerous in a way that a much bigger one is not, so scaling up out of a bad region is a legitimate move rather than an admission of defeat. Second, early stopping can be the wrong instinct, because epoch-wise double descent means test error can rise and then fall again with more training. Stopping at the first rise leaves performance on the table.

Neither implies that scale is free. Double descent describes the shape of the curve, not its level, and scaling laws describe how the level moves with compute. A model in the modern regime still obeys them. See also grokking, which is arguably epoch-wise double descent in a regime where the second descent takes a hundred thousand steps.