Fisher Information and Natural Gradient
Why the steepest descent direction depends on the coordinate system you happened to choose, how the Fisher matrix fixes that, and which popular optimisers are approximating it badly.
Gradient descent takes the steepest step in parameter space, measuring distance with the Euclidean norm on \(\theta\). That is an arbitrary choice. Rescale a layer's weights by 10 and divide its outputs by 10 and the model computes exactly the same function, yet the gradient, and therefore the trajectory, changes completely. The geometry that matters for a probabilistic model is not distance between parameters but distance between the distributions those parameters define.
The metric that distributions actually live in
Take a model \(p_\theta(y \mid x)\) and ask how far \(p_{\theta + \delta}\) is from \(p_\theta\) in KL divergence. Expand to second order; the first-order term vanishes because \(\theta\) is a stationary point of the KL from itself, leaving
where the Fisher information matrix is
\(F\) is the local metric tensor on the statistical manifold. Steepest descent under that metric, rather than under the Euclidean one, gives the natural gradient \(F^{-1} \nabla_\theta \mathcal{L}\), which is invariant to smooth reparameterisation: the update moves the distribution the same way regardless of how the model was coordinatised. The Fisher also equals the negative expected Hessian of the log-likelihood, which is why it acts like curvature without ever being indefinite.
Making it computable
\(F\) is \(d \times d\) for \(d\) parameters, so forming it, let alone inverting it, is hopeless past a few thousand parameters. Every practical method is a structured approximation.
K-FAC treats the block of \(F\) for a whole layer as a Kronecker product of two much smaller matrices, one from the layer's inputs and one from its output gradients. The result is neither diagonal nor low rank yet inverts cheaply, and costs only a small multiple of plain SGD per step (Martens and Grosse, ICML 2015, arXiv:1503.05671).
Diagonal approximations are what most people actually run. Adam's per-parameter second-moment estimate is a diagonal preconditioner built from squared gradients, which is often described as a diagonal empirical Fisher. The description deserves care: the empirical Fisher uses gradients of the observed labels rather than samples from the model, and it does not generally capture second-order information, with the conditions for it to approximate the true Fisher unlikely to hold in practice (Kunstner, Balles and Hennig, NeurIPS 2019, arXiv:1905.12558). Adam works; the natural-gradient story told about why it works is shakier than usually admitted.
Importance weighting. Elastic weight consolidation uses the diagonal Fisher of an old task to decide which weights are expensive to move, adding a quadratic penalty that slows learning on exactly those coordinates (Kirkpatrick et al., PNAS 114(13), 2017, arXiv:1612.00796). Here the Fisher is not being used to accelerate optimisation at all; it is being used as a measure of parameter importance, which is the same object doing different work.
Trust-region policy optimisation is the other landmark use: constrain the policy update by KL, which by the expansion above is a quadratic form in \(F\), and the constrained step is a natural gradient scaled to hit the KL budget.
When it breaks
The inverse amplifies the worst-estimated directions. Small Fisher eigenvalues correspond to directions the data says little about, and \(F^{-1}\) multiplies those by a large number. Damping, adding \(\gamma I\) before inverting, is mandatory, and its value matters as much as the learning rate.
Estimation cost scales with the noise you want removed. \(F\) is an expectation over model samples. Estimating it from a minibatch gives a rank-limited, noisy matrix, and stale estimates refreshed every \(k\) steps are standard, which means the metric is always slightly wrong.
Invariance is a property of the exact update. Reparameterisation invariance holds for the true natural gradient, not for a damped, block-diagonal, staled approximation. Most of the theoretical appeal evaporates in the approximation, which is a fair summary of why second-order methods have not displaced Adam at scale.
12 flashcards for this concept
Click a card to reveal the answer.