Spectral Analysis of Weight Matrices
What the singular values of a weight matrix control, why random matrix theory predicts the spectrum at initialisation, and how spectral thinking connects initialisation, LoRA and modern optimisers.
A weight matrix is a linear map, and everything a linear map does to signal magnitude is contained in its singular values. Write \(W = U \Sigma V^\top\); the largest singular value \(\sigma_{\max}\) is the most any input direction can be stretched, and it is exactly the Lipschitz constant of the layer. Whether a hundred-layer network's activations explode, vanish or hold steady is a statement about products of these spectra, not about any individual weight.
What the spectrum looks like before training
At initialisation, weights are drawn i.i.d., which puts them squarely inside random matrix theory. For an \(m \times n\) matrix with i.i.d. entries of variance \(\sigma^2 / n\) and aspect ratio \(q = m/n \le 1\), the empirical distribution of singular values converges to the Marchenko-Pastur law, supported on
Two consequences follow immediately. Singular values are not concentrated at one value; even a perfectly conditioned-looking Gaussian init has a spread that widens as the matrix becomes less square. And the top singular value is predictable, which is what makes He and Glorot initialisation derivable rather than empirical: choose \(\sigma\) so the expected squared singular value is 1 and signal variance is preserved layer to layer.
The sharper statement is about the input-output Jacobian of the whole network. Dynamical isometry is the condition that all of its singular values concentrate near 1, and networks that achieve it learn orders of magnitude faster. Free-probability analysis of the full singular value distribution shows ReLU networks are incapable of dynamical isometry at any initialisation, while sigmoidal networks achieve it with orthogonal weights and not with Gaussian ones (Pennington, Schoenholz and Ganguli, NeurIPS 2017, arXiv:1711.04735). Mean-field variance preservation is a statement about the average singular value; dynamical isometry is a statement about the whole distribution, and the difference is what the deeper analysis buys.
Spectral thinking after initialisation
Effective rank. Trained weight updates are often close to low rank, and the stable rank \(\lVert W \rVert_F^2 / \sigma_{\max}^2\) quantifies it without a hard threshold. This is the empirical fact LoRA monetises: if \(\Delta W\) from fine-tuning has a rapidly decaying spectrum, parameterising it as \(BA\) with small inner dimension loses little.
Spectral norm as a control surface. Bounding \(\sigma_{\max}\) bounds the layer's Lipschitz constant, which is the mechanism behind spectral normalisation in GAN discriminators and behind generalisation bounds stated in terms of products of spectral norms. It is also the reason attention logit growth is controllable by normalising queries and keys rather than by clipping the output.
Conditioning of the update, not the weight. Recent optimisers act on the spectrum of the gradient: orthogonalising an update matrix sets all of its singular values to 1, so every direction receives comparable movement instead of the step being dominated by the top singular direction. That is a spectral prescription, and it explains why such methods behave differently from any diagonal preconditioner, which cannot see singular directions at all.
Where the analysis stops working
Products of dependent matrices. Layer-wise Marchenko-Pastur assumes independence. After training, adjacent layers are correlated through the loss, so composing per-layer spectra to predict the network Jacobian is a heuristic, not a theorem.
Heavy tails appear with training. Initialised spectra are bulk-only. Trained spectra grow outliers, a large-eigenvalue tail separated from the bulk, and it is those spikes rather than the bulk that carry the learned structure. Diagnostics calibrated on random-matrix bulk behaviour misread trained networks.
Spectral norm is a worst case. The Lipschitz bound assumes an adversarial input aligned with the top singular vector. Real activations are not adversarial, so bounds built from products of spectral norms are true and often vacuous, off by many orders of magnitude at realistic depth.
Cost. A full SVD of a \(d \times d\) matrix is \(O(d^3)\). Monitoring spectra of every layer during training is not free, which is why practical diagnostics use power iteration for \(\sigma_{\max}\) and stochastic estimators for the rest.
12 flashcards for this concept
Click a card to reveal the answer.