The Neural Tangent Kernel and Feature Learning

If you want a theory of deep learning, the obstacle is non-convexity: gradient descent on a network is an optimisation over a landscape nobody can characterise. Jacot, Gabriel, and Hongler found a limit in which the problem becomes convex and completely solvable, and the limit is width going to infinity (Jacot et al., 2018, Neural Tangent Kernel, arXiv:1806.07572, NeurIPS 2018).

The result is one of the cleanest in the field, and its most useful consequence is negative: it tells you exactly which part of deep learning it fails to explain.

The linearisation

Write the network output as \(f(x; \theta)\). First-order Taylor expansion around initialisation \(\theta_0\):

This is linear in \(\theta - \theta_0\), so training it with squared loss is kernel regression with kernel

The neural tangent kernel. The approximation is only useful if it stays valid, which requires \(\theta\) to remain near \(\theta_0\) throughout training. Jacot et al. proved that under the right parameterisation this is exactly what happens as width \(\to \infty\): the kernel converges to a deterministic limit at initialisation, and it stays constant during training. Every parameter moves a vanishing amount; there are just enormously many of them.

So an infinitely wide network trained by gradient descent is a kernel machine with a kernel fixed before it sees any data. The dynamics are closed-form, convergence to global minimum is guaranteed, and generalisation is standard kernel theory.

Lazy and rich

The regime has a name that captures the objection: lazy training (Chizat et al., 2019, arXiv:1812.07956). The features, meaning the gradient map \(\nabla_\theta f(\cdot; \theta_0)\), never change. The network fits data by re-weighting features it was born with.

Real networks do not behave this way, and the difference is the entire point of deep learning. A trained ResNet's early layers detect edges; at initialisation they detected noise. A trained transformer has induction heads; at initialisation it did not. The rich or feature-learning regime is where representations change, and it is empirically stronger: finite-width networks routinely beat their own infinite-width NTK on the same task, sometimes by large margins.

Which regime you land in is not fixed by architecture. It is controlled by parameterisation and output scale. Standard parameterisation with growing width drifts toward lazy; the maximal-update parameterisation (\(\mu\)P) is constructed specifically so that features keep learning as width grows (Yang & Hu, 2021, arXiv:2011.14522). This is the theoretical foundation under \(\mu\)P and hyperparameter transfer: \(\mu\)P works because it keeps the infinite-width limit in the rich regime, where the optimal learning rate becomes width-independent.

The lazy-to-rich crossing also gives one of the cleanest accounts of grokking, where a network first fits data lazily and later transitions to feature learning (Kumar et al., 2024, arXiv:2310.06110).

What the NTK is good for

Dismissing it would be a mistake. Three things it does well:

It explains why over-parameterised networks train at all. In the NTK regime the loss is convex in the linearised parameters and gradient descent provably reaches a global minimum. The mystery of optimising a non-convex objective partly dissolves: near the lazy regime, the objective is nearly convex.

It gives a spectral account of learning order. Kernel regression fits the top eigendirections of \(\Theta\) first. Since the NTK's spectrum is biased toward low-frequency functions, this predicts the observed spectral bias, where networks learn smooth structure before high-frequency detail. That prediction holds well outside the regime that produced it.

It is a computable baseline. The NTK of an architecture can be evaluated exactly, giving a well-defined "what you get without feature learning" reference. Any gain over it is attributable to feature learning, which turns a vague claim into a measurement.

When it breaks

Quantitatively, the width required for the kernel to stay constant is far beyond anything trained in practice, and the deviation is not a small correction: the finite-width kernel evolves substantially, and that evolution is the learning that matters.

Qualitatively, the regime cannot express transfer learning, since features that never change cannot transfer. It cannot express the benefit of depth, since the kernel of a deep network is just a different kernel, not a fundamentally more capable object. And it says nothing about the mechanisms interpretability research finds, because circuits are features and the NTK has none.

The honest summary is that the NTK explains why training works and fails to explain why deep learning works. Both halves are worth knowing, and the second is why the field's theory effort has moved toward feature learning, \(\mu\)P, and the spectral analysis of weight matrices.