The Bias-Variance Tradeoff Is a Special Case: Double Descent, Benign Overfitting, and Grokking

In 2016 a group at Google Brain and MIT took a standard image classifier, replaced every training label with a random one, and trained to convergence. The network reached zero training error. It had memorised a million arbitrary label assignments with no structure to exploit, which meant its effective capacity exceeded the dataset entirely. Then they trained the same architecture on the real labels, and it generalised as well as it always had (Zhang et al., 2017, Understanding deep learning requires rethinking generalization, arXiv:1611.03530, ICLR 2017).

Under the account in every machine learning textbook of the preceding twenty-five years, this is impossible. A hypothesis class rich enough to fit arbitrary labels has, by definition, no useful generalisation guarantee. The bound is not merely loose; it is vacuous, promising a test error of at most 100 percent. Yet the same class, trained by the same algorithm on real data, produces models that work.

The resolution took most of a decade and arrived as three apparently unrelated phenomena. Double descent showed that test error is not U-shaped in capacity. Benign overfitting characterised when interpolating noisy data is harmless. Grokking showed that generalisation can appear a hundred thousand steps after the training loss went flat. They look like three separate curiosities. They are three views of one shift: the object that controls generalisation is not the hypothesis class, it is the algorithm's implicit bias over that class.

Why this matters: Every practical instinct built on the U-curve is now conditional. "The model is overfitting, make it smaller" can be exactly backwards. "Validation error is rising, stop training" can leave a second descent unclaimed. "More data always helps" is false in a measurable regime. If you tune models, these are not philosophical points; they change what you do on a Tuesday afternoon.

TL;DR

• Fitting training data exactly is not the failure mode classical theory said it was, and it is now the default operating regime for every frontier model.
• Test error against capacity has a peak at the interpolation threshold, where capacity just suffices to fit the data, and falls again past it. The classical U-curve is the left half of this longer curve (Belkin et al., 2019, PNAS).
• In the isotropic ridgeless regression model the risk diverges at \(\gamma = p/n = 1\) and, when the signal-to-noise ratio exceeds 1, has a genuine interior minimum in the over-parameterised region (Hastie et al., 2022, Annals of Statistics).
• Double descent also appears in epochs and in dataset size. For a fixed model, adding training data can raise test error, if it moves you toward the threshold from the over-parameterised side (Nakkiran et al., 2020).
• Overfitting is benign when the data covariance has many low-variance directions to absorb noise while signal directions are fitted cleanly. It is a property of the data spectrum, not a universal law (Bartlett et al., 2020, PNAS).
• Grokking is delayed generalisation on a flat loss curve: memorisation, then silent circuit formation, then weight-decay cleanup that exposes the generalising circuit (Power et al., 2022; Nanda et al., 2023).
• The unifying claim is that the optimiser selects the solution. Among the infinitely many interpolants, gradient descent finds low-norm, low-complexity ones, and every bound that ignores the algorithm is bounding the wrong thing.
• No current theory predicts observed generalisation well enough to guide design. This is an explanation, not yet an engineering tool.

At a Glance

flowchart LR
    D["Training data<br/>n examples"] --> C{"Model capacity"}
    C -->|"Under-parameterised"| U["Classical regime<br/>U-shaped test error<br/>bias vs variance"]
    C -->|"At the threshold"| T["Interpolation threshold<br/>one interpolant, forced to be wild<br/>test error peaks"]
    C -->|"Over-parameterised"| O["Modern regime<br/>many interpolants exist"]
    O --> S["Optimiser picks one<br/>implicit bias toward low norm"]
    S --> B{"Does the data spectrum<br/>have room for the noise?"}
    B -->|"Yes"| BE["Benign overfitting<br/>test error falls again"]
    B -->|"No"| CA["Catastrophic overfitting<br/>classical theory was right"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff

    class D blue
    class C,B purple
    class U teal
    class T amber
    class O,S purple
    class BE emerald
    class CA rose

Before the U-Curve Broke

The classical account is genuinely beautiful, which is part of why it survived contradiction for so long. Decompose the expected squared error of an estimator into three terms:

Richer models reduce bias and raise variance. Somewhere the sum has a minimum. Geman, Bienenstock, and Doursat wrote the canonical statement of this for neural networks, arguing that the dilemma was fundamental and that neural networks would need strong architectural priors to escape it (Geman et al., 1992, Neural Computation 4(1):1-58). Vapnik and Chervonenkis had already supplied the machinery to make "richer" precise, and VC dimension became the standard capacity measure.

timeline
    title From the Dilemma to the Modern Regime
    1992 : Geman, Bienenstock and Doursat state the bias/variance dilemma for neural networks
         : Capacity control becomes the organising principle of model selection
    2016 : Zhang et al. fit ImageNet with random labels to zero training error
         : Uniform-convergence bounds are shown to be vacuous for real networks
    2018 : Belkin et al. describe double descent
         : Garipov and Draxler independently find low-loss paths between minima
    2019 : Nakkiran et al. confirm double descent in epochs and in sample size
         : Effective model complexity proposed as the governing variable
    2020 : Bartlett, Long, Lugosi and Tsigler characterise benign overfitting
         : Effective rank of the covariance decides whether interpolation is safe
    2021 : Cohen et al. show gradient descent trains at the edge of stability
         : Curvature is negotiated by the optimiser, not fixed by the problem
    2022 : Power et al. report grokking on algorithmic datasets
         : Generalisation arrives long after the training loss stops moving
    2023 : Nanda et al. reverse-engineer the grokked circuit
         : Varma et al. explain the delay through circuit efficiency
    2024 : Kumar et al. frame grokking as a lazy-to-rich transition
         : Feature learning, not capacity, becomes the operative variable

Zhang et al.'s random-label experiment is the hinge. It was not the first hint that something was wrong, but it was the first result that could not be explained away, because it demolished a specific argument rather than merely reporting an anomaly. If a class can fit arbitrary labels, its Rademacher complexity is near maximal, so the uniform bound

returns a number greater than one. The theory does not merely underestimate; it says nothing at all.

[IMAGE: Two-panel figure. Left: classical U-curve of test error against capacity with bias and variance components dashed underneath. Right: the same axes extended to the right, showing the peak at the interpolation threshold and the second descent, with the classical U shaded to show it is the left portion. Caption: "The textbook picture is not wrong; it is cropped."]

What Actually Determines Generalisation

The peak, and why it is there

Belkin, Hsu, Ma, and Mandal plotted test error against capacity past the point where classical practice stops looking, and found the curve does not keep rising (Belkin et al., 2019, PNAS 116(32):15849-15854). It peaks at the interpolation threshold, where capacity is just sufficient to fit the training data exactly, and then falls.

The mechanism is a counting argument. Below the threshold, no model fits the data exactly, and error is governed by the usual approximation-estimation trade. At the threshold there is essentially a unique interpolating solution, and the fit is completely determined by the data including its noise, with no freedom left to be well-behaved anywhere the data does not constrain it. Above the threshold there are infinitely many interpolating solutions, and the question becomes which one you get.

That last sentence is the whole shift. Capacity stops being the controlling variable and selection takes over.

[IMAGE: Three panels showing fitted curves through 12 noisy points for a polynomial basis of degree 8, 12, and 60, all using the minimum-norm solution. Degree 12 (the interpolation threshold) oscillates violently between points; degree 60 passes through every point smoothly. Caption: "Same data, same interpolation constraint. More freedom to interpolate means freedom to do it smoothly."]

The optimiser is the regulariser

For linear regression, gradient descent from zero initialisation converges to the minimum \(\ell_2\)-norm interpolant. Nobody asked for that; it is a property of the algorithm's trajectory, and it is the simplest example of an implicit bias. More capacity means more interpolants to choose from, so the minimum-norm one gets smoother, not wilder. The second descent follows.

Deep networks have implicit biases too, less cleanly characterised: gradient descent on separable data drives linear classifiers toward the max-margin solution, and training at the edge of stability biases the trajectory toward flatter regions without any explicit sharpness penalty (Cohen et al., 2021, arXiv:2103.00065). The common structure is that the algorithm, not the class, picks the answer, and the bound that ignores the algorithm cannot see it.

When interpolation is safe, precisely

Bartlett, Long, Lugosi, and Tsigler answered the sharp version of the question for linear regression: exactly when does a minimum-norm interpolator of noisy data generalise? Their characterisation is in terms of two notions of effective rank of the data covariance (Bartlett et al., 2020, PNAS 117(48):30063-30070).

The intuition is worth stating in plain terms. An interpolator must fit the noise; it has no choice. The question is where the noise goes. If the covariance spectrum has a small number of high-variance directions carrying signal and a large number of low-variance directions carrying almost nothing, the interpolator can absorb the noise into that long tail, where it contributes negligibly to prediction on new points. If the spectrum decays too quickly, there is no such tail, and the noise must be absorbed into directions that matter.

[IMAGE: Two eigenvalue spectra plotted as sorted bar charts on a log-y axis. Left: fast decay, a handful of large eigenvalues and nothing beyond, labelled "nowhere to put the noise". Right: a few large eigenvalues followed by a long slowly-decaying tail, labelled "benign". Under each, the noise vector drawn as arrows landing in the tail or in the head. Caption: "Interpolation must fit the noise; the spectrum decides where it lands."]

Benign overfitting is therefore a property of the data, not a law of over-parameterisation. High-dimensional natural data tends to have the required structure; low-dimensional tabular data frequently does not, which is why gradient-boosted trees still win there and why "just make it bigger" is domain-specific advice masquerading as a principle.

Grokking: the same story on the time axis

Power et al. trained small transformers on modular arithmetic and watched training accuracy hit 100 percent within a few hundred steps while test accuracy stayed at chance for tens of thousands more, before rising abruptly to 100 percent (Power et al., 2022, arXiv:2201.02177).

Nanda et al. reverse-engineered what was happening and found the network had built a discrete Fourier circuit, embedding inputs on a circle and using trigonometric identities to convert addition into rotation (Nanda et al., 2023, arXiv:2301.05217, ICLR 2023). Their progress measures split training into three phases that are continuous even where the loss curve is flat.

stateDiagram-v2
    [*] --> Memorisation
    Memorisation: Lookup circuit forms
    Memorisation: Train loss falls, test loss flat
    Memorisation --> CircuitFormation: train loss bottoms out
    CircuitFormation: Fourier circuit grows in parallel
    CircuitFormation: Both loss curves flat, progress measures rising
    CircuitFormation --> Cleanup: generalising circuit is competitive
    Cleanup: Weight decay removes the redundant memorising circuit
    Cleanup: Test accuracy snaps to 100 percent
    Cleanup --> [*]

Varma et al. explained the delay: both a memorising and a generalising solution exist, the generalising one produces larger logits per unit parameter norm and is therefore more efficient under weight decay, but it is slower to learn. Their theory predicted, before anyone had observed them, ungrokking (a grokked network regressing when retrained on a dataset below a critical size) and semi-grokking (partial generalisation at intermediate dataset size), and both were confirmed (Varma et al., 2023, arXiv:2309.02390).

Two more accounts converge on the same structure from different directions. Liu et al. showed the operative variable is the weight norm: train loss against norm looks like an "L" and test loss like a "U", so a large initialisation starts on the wrong side and must walk to the optimum, and shrinking the initialisation removes grokking on MNIST entirely (Liu et al., 2023, Omnigrok, arXiv:2210.01117). Kumar et al. framed it as the crossing from lazy (kernel) dynamics to rich feature learning (Kumar et al., 2024, arXiv:2310.06110, ICLR 2024).

All four say: two solutions race, one is faster to reach, one is favoured by the regulariser, and the delay is the crossover time. Which is exactly epoch-wise double descent, observed in a regime where the second descent takes a hundred thousand steps.

Seeing It in Motion

graph TD
    Q["Why did this model generalise?"] --> A["Classical answer<br/>the hypothesis class was small enough"]
    Q --> B["Modern answer<br/>the algorithm selected a good interpolant"]
    B --> B1["Implicit bias<br/>min norm, max margin, flat minima"]
    B --> B2["Data spectrum<br/>somewhere harmless to put the noise"]
    B --> B3["Feature learning<br/>rich regime, not lazy"]
    A --> A1["Falsified by random-label fitting"]
    B1 --> R["Bounds must be algorithm-dependent"]
    B2 --> R
    B3 --> R
    R --> P1["PAC-Bayes over the reached posterior"]
    R --> P2["Compression-based bounds"]
    R --> P3["Norm and margin based bounds"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

    class Q purple
    class A,A1 rose
    class B,B1,B2,B3 blue
    class R teal
    class P1,P2,P3 emerald

The diagram makes the disagreement concrete. Every arrow on the right-hand side depends on the training procedure. Every arrow on the left depends only on the class. That is the substance of the shift, and it explains why the classical bounds cannot be repaired by tightening constants.

[IMAGE: Three small multiples on shared axes, all showing the same double-descent shape on different x-axes: model width, training epochs, and training set size. Mark the interpolation threshold on each. Caption: "One phenomenon, three axes. Nakkiran et al.'s effective model complexity is the variable that makes all three the same plot."]

By the Numbers

Isotropic ridgeless regression is the cleanest setting where the whole curve is available in closed form. Writing \(\gamma = p/n\) for the ratio of features to samples, \(\sigma^2\) for the noise variance, and \(r^2 = \lVert\beta\rVert^2\) for the signal strength, the asymptotic out-of-sample risk of the minimum-norm interpolator is

(Hastie, Montanari, Rosset & Tibshirani, 2022, Surprises in High-Dimensional Ridgeless Least Squares Interpolation, Annals of Statistics 50(2):949-986).

Values computed directly from the formula above with \(\sigma^2 = 1\). The interior minimum exists only when SNR exceeds 1, at \(\gamma^{\ast} = (1 - \sigma/r)^{-1}\), which for \(r^2 = 5\) gives \(\gamma^{\ast} = 1.81\).

Two features of the table deserve attention. The divergence at \(\gamma = 1\) is not a numerical artefact; the risk genuinely blows up, and it is the sharpest possible statement that a model just large enough to interpolate is the worst choice available. And at SNR = 5 there is a genuine local minimum at \(\gamma = 1.81\) with risk 3.47, comfortably below the null risk of 5, so over-parameterisation is doing real work rather than merely recovering.

The honest caveat is that the under-parameterised branch of this particular model carries no bias term by construction, so the two branches are not directly comparable as nested models. This toy shows the peak and the second descent cleanly and does not by itself show over-parameterisation beating the classical optimum. Getting that requires the anisotropic spectrum of the benign-overfitting analysis, which is precisely why Bartlett et al.'s effective-rank conditions are the load-bearing result rather than this closed form.

A Concrete Example

Take \(n = 200\) training samples, noise variance \(\sigma^2 = 1\), and signal strength \(r^2 = 5\), so the signal-to-noise ratio is 5. Sweep the number of features \(p\) and read the risk off the formula.

Step 1: \(p = 100\), so \(\gamma = 0.5\). Under-parameterised. Risk is \(\sigma^2\gamma/(1-\gamma) = 1 \times 0.5 / 0.5 = 1.00\). Ordinary least squares with twice as many samples as parameters, behaving exactly as any statistics course would predict.

Step 2: \(p = 180\), so \(\gamma = 0.9\). Still under-parameterised, but the design matrix is becoming ill-conditioned. Risk is \(0.9 / 0.1 = 9.00\). The model has not changed in any way a practitioner would notice; the risk is nine times worse.

Step 3: \(p = 198\), so \(\gamma = 0.99\). Risk is \(0.99/0.01 = 99.0\). Classical theory called this variance explosion and it was right. This is the regime where every instinct says "the model is too big".

Step 4: \(p = 200\), so \(\gamma = 1\). The interpolation threshold. Risk diverges. There is a unique solution passing through every training point and it is determined entirely by the data, noise included.

Step 5: \(p = 220\), so \(\gamma = 1.1\). Now over-parameterised. Risk is \(5(1 - 1/1.1) + 1/0.1 = 0.455 + 10 = 10.45\). Still terrible, but finite, and the trend has reversed.

Step 6: \(p = 362\), so \(\gamma = 1.81\). Risk is \(5(1 - 0.553) + 1/0.809 = 2.24 + 1.24 = 3.47\). This is the minimum over the over-parameterised branch, and it is 28 times better than the model at \(p = 198\), which had fewer parameters.

Step 7: \(p = 2000\), so \(\gamma = 10\). Risk is \(5(0.9) + 1/9 = 4.50 + 0.11 = 4.61\). Rising again toward the null risk of 5 as the fixed signal is spread over more and more coordinates.

The shape a practitioner should carry away from those seven steps: the risk at \(p=198\) is worse than at \(p=362\), and both are worse than at \(p=100\). Someone tuning \(p\) by bisection from below, watching validation error climb from 1.00 to 99.0, would conclude the model class was hopeless and stop. Someone who pushed through would find a usable model at nearly twice the capacity. This is not a hypothetical failure mode; it is what "our bigger model got worse so we stopped scaling" looks like from the inside.

[IMAGE: The seven-step sweep plotted as risk against p on a log-y axis, with each step annotated and the divergence at p=200 drawn as a vertical asymptote. Shade the region 180 to 260 in red and label it "the trap". Caption: "Bisecting from below finds the trap and stops."]

Where It Breaks

The peak needs noise to be visible

Nakkiran et al. found the height of the double-descent peak grows sharply with the fraction of corrupted training labels, and on clean data it can be barely detectable (Nakkiran et al., 2020, Deep Double Descent, arXiv:1912.02292, ICLR 2020). A practitioner who has never observed double descent has probably never trained near the threshold on noisy labels, which is not the same as having disproved it. Conversely, someone who sees a large peak should suspect label noise before suspecting anything else.

[IMAGE: Family of double-descent curves for the same architecture at label-noise fractions 0%, 5%, 10%, 20%, showing the peak at the interpolation threshold growing from barely visible to dominant. Caption: "The peak's height is a measurement of your labels, not only of your architecture."]

Implicit regularisation is characterised for models nobody uses

The clean results, minimum norm for linear regression, max margin for separable linear classification, are theorems about linear models. For deep networks the implicit bias is architecture-dependent, optimiser-dependent, initialisation-dependent, and in most settings not known in closed form. Adam has a different implicit bias from SGD, which is one of several reasons Adam-trained models generalise differently, and the literature has no complete account of either.

"Capacity" is not parameter count

Nakkiran et al.'s effective model complexity, defined as the largest number of samples the whole training procedure can drive to near-zero error, is a property of architecture plus optimiser plus training duration plus data. Parameter count is a poor proxy, which is why published double-descent curves plotted against width sometimes fail to reproduce when someone changes the schedule. If you go looking for the threshold, you are looking for a property of your pipeline and not of your architecture.

The bounds are still vacuous, and the repairs are not free

PAC-Bayes bounds computed over the posterior a training procedure actually reaches have produced the first genuinely non-vacuous numerical bounds for real networks, which is a real achievement. They remain far too loose to guide a design decision: knowing test error is below 40 percent does not help when you measured 8 percent. Compression-based and margin-based bounds have the same character. None of this is a solved problem, and papers claiming otherwise are usually bounding a modified network or a modified training procedure.

Grokking does not scale up

Grokking is demonstrated cleanly on small algorithmic datasets with a single underlying rule, heavy weight decay, and near-full-batch training. Frontier pretraining has none of those properties: no single rule, roughly one pass over each example, and no memorise-then-generalise structure because there is nothing to memorise twice. Reasoning from grokking to LLM training dynamics is an analogy, and it should be labelled as one.

What does transfer is methodological. A flat loss curve is not evidence that learning has stopped, and if you rely on training loss to decide when to stop, you will occasionally stop in the middle of circuit formation.

Selection on the validation set re-creates the problem

Everything above is about a single training run. Choosing the best of twenty checkpoints against one held-out set is itself a search over hypotheses using the data, and it re-introduces exactly the multiple-comparison problem that uniform bounds were invented for. The over-fitting has simply moved up a level, into the model selection procedure, where nobody is measuring it.

Alternative Designs

The table has one honest column and it is the last one. No framework here predicts the generalisation gap of a model you are about to train, which is why the field's actual practice, scaling laws fitted empirically to loss, is not theory-driven and does not need to be.

How It Is Used in Practice

Scale through the trap, do not stop at it. The most direct consequence is that a model performing worse than a smaller one is not evidence that the class is exhausted. If the degradation appeared as capacity approached the point where training error hits zero, more capacity is the indicated move rather than the forbidden one. This is uncomfortable advice because it is exactly what a naive practitioner also says.

Do not early-stop on the first rise. Epoch-wise double descent means validation error can rise and fall again. Where compute permits, train past the first rise before concluding, and keep the checkpoint from before it as insurance. The cost is compute; the alternative is silently discarding a better model.

Regularise deliberately, not by shrinking. Weight decay, label smoothing, and data augmentation change which interpolant the optimiser selects, which is the actual lever. Reducing capacity moves you toward the threshold, which is the wrong direction from the over-parameterised side.

[IMAGE: A grokking training run drawn as four stacked time series on a shared log-x axis: train accuracy, test accuracy, weight norm, and a restricted-loss progress measure. Train accuracy saturates first; the progress measure rises steadily through the flat region; weight norm falls; test accuracy snaps up last. Caption: "Four views of one run. Only one of them is flat during the interesting part."]

Track progress measures, not just loss. The grokking literature's transferable contribution is the habit of instrumenting training with quantities tied to the mechanism you care about: probe accuracy, circuit norms, an unsaturated downstream eval. Loss is one projection of a high-dimensional process and it is not always the informative one.

Assume label noise is present, because it is. The peak's height scales with label noise, and every large scraped dataset has some. If your capacity sweep shows a pronounced peak, that is a signal about your labels as much as about your architecture.

Insights Worth Remembering

1. The classical theory is not wrong, it is cropped. Everything in the U-curve is correct in the under-parameterised regime, which was the only regime anyone could afford to train in for fifty years. The error was universalising a special case, and the lesson generalises beyond this instance.

2. The interpolation threshold is the most dangerous place to be, and it is where "make it smaller" sends you. A model just big enough to fit the data has a unique interpolant that is fully determined by the noise. Both directions away from it are improvements.

3. Over-parameterisation is not what makes interpolation safe; the data spectrum is. Benign overfitting requires many low-variance directions in which to bury the noise. High-dimensional natural data usually has them and low-dimensional tabular data usually does not, which is the real reason "just scale it" is domain-specific advice.

4. Generalisation is a property of the algorithm, not the hypothesis class. Any bound that does not mention the optimiser is bounding the worst element of a set that the optimiser will never return. That is why tightening constants was never going to fix uniform convergence.

5. A flat loss curve does not mean nothing is happening. Grokking's real contribution is not the phenomenon, which does not scale, but the demonstration that the metric everyone watches can be flat through the most important phase of training.

6. Double descent in epochs makes early stopping a heuristic rather than a principle. It is a good heuristic under a compute budget, and it should be recognised as a budget decision rather than a statistical one.

7. The theory is explanatory and not yet predictive, and that gap is the state of the field. We can say why a trained model generalised. We cannot say, before training, how well it will. Practice has routed around this with empirical scaling laws, and the routing has held for six years.

Open Questions

Can a bound become tight enough to be useful? PAC-Bayes has produced non-vacuous numerical bounds for real networks, which was thought impossible in 2016. Measured gaps remain many times smaller than bounded ones. Whether the remaining looseness is a technical artefact or something fundamental about worst-case analysis is genuinely open, and the answer determines whether generalisation theory ever becomes an engineering tool.

What is the implicit bias of Adam? The min-norm and max-margin results are for gradient descent. Adam is what everyone actually uses, its implicit bias is not characterised in general, and the empirical differences in the solutions it finds are large and reproducible. This is a well-posed question that has resisted a clean answer.

Does anything like double descent occur in LLM pretraining? Frontier training runs sit far from the interpolation threshold, seeing each token roughly once, so the standard picture may simply not apply. Whether a data-repetition analogue exists, and whether it interacts with observed loss plateaus, is measurable and largely unmeasured at scale.

Is grokking a special case of something general, or a small-data artefact? The circuit-efficiency account is theoretically appealing and made confirmed novel predictions, which is strong evidence for the mechanism in the regime studied. Whether competing circuits with different learning speeds and different efficiencies is a general description of neural network training, or a description of what happens when a small network faces a clean algorithmic rule, is unresolved.

What is the right complexity measure? Parameter count fails. Norm-based measures capture some of it. Effective model complexity is defined operationally rather than analytically. Singular learning theory offers the local learning coefficient as a candidate grounded in the geometry of degenerate parameter spaces. As of early 2026, no measure predicts generalisation across architectures well enough to select between two models without training both.

Sources and Further Reading

1. Geman, S., Bienenstock, E., & Doursat, R. (1992). "Neural Networks and the Bias/Variance Dilemma." Neural Computation, 4(1), 1-58. DOI:10.1162/neco.1992.4.1.1
2. Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2017). "Understanding deep learning requires rethinking generalization." ICLR. arXiv:1611.03530
3. Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). "Reconciling modern machine-learning practice and the classical bias-variance trade-off." PNAS, 116(32), 15849-15854. arXiv:1812.11118
4. Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2020). "Deep Double Descent: Where Bigger Models and More Data Hurt." ICLR. arXiv:1912.02292
5. Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020). "Benign overfitting in linear regression." PNAS, 117(48), 30063-30070. DOI:10.1073/pnas.1907378117
6. Hastie, T., Montanari, A., Rosset, S., & Tibshirani, R. J. (2022). "Surprises in High-Dimensional Ridgeless Least Squares Interpolation." Annals of Statistics, 50(2), 949-986. arXiv:1903.08560
7. Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets." arXiv:2201.02177
8. Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2023). "Progress measures for grokking via mechanistic interpretability." ICLR. arXiv:2301.05217
9. Varma, V., Shah, R., Kenton, Z., Kramár, J., & Kumar, R. (2023). "Explaining grokking through circuit efficiency." arXiv:2309.02390
10. Liu, Z., Michaud, E. J., & Tegmark, M. (2023). "Omnigrok: Grokking Beyond Algorithmic Data." ICLR. arXiv:2210.01117
11. Kumar, T., Bordelon, B., Gershman, S. J., & Pehlevan, C. (2024). "Grokking as the Transition from Lazy to Rich Training Dynamics." ICLR. arXiv:2310.06110
12. Cohen, J. M., Kaur, S., Li, Y., Kolter, J. Z., & Talwalkar, A. (2021). "Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability." ICLR. arXiv:2103.00065
13. Barak, B., Edelman, B. L., Goel, S., Kakade, S., Malach, E., & Zhang, C. (2022). "Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit." NeurIPS. arXiv:2207.08799
14. Jacot, A., Gabriel, F., & Hongler, C. (2018). "Neural Tangent Kernel: Convergence and Generalization in Neural Networks." NeurIPS. arXiv:1806.07572
15. Bartlett, P. L., & Mendelson, S. (2002). "Rademacher and Gaussian Complexities: Risk Bounds and Structural Results." JMLR, 3, 463-482. jmlr.org/papers/v3/bartlett02a.html

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.