PCA, SVD and Whitening

Principal component analysis is usually introduced twice, as "find the directions of maximum variance" and as "find the best low-rank approximation", with no explanation of why two different-sounding goals produce identical answers. They are the same optimisation seen from opposite sides, and the Pythagorean identity connecting them is the cleanest way in.

For centred data, total variance splits exactly into the variance captured by a projection plus the variance in the residual. Maximising the first therefore minimises the second; there is no trade-off and no second criterion to balance. Everything else about PCA follows from that.

Two derivations, one answer

Directly, the first principal direction solves \(\max_{\|w\|=1} w^\top \Sigma w\) where \(\Sigma = \frac{1}{n}X^\top X\) is the sample covariance. The Lagrangian gives \(\Sigma w = \lambda w\), so \(w\) is an eigenvector of \(\Sigma\) and \(\lambda\) is the variance along it. Subsequent components solve the same problem restricted to the orthogonal complement, so the components are the eigenvectors ordered by eigenvalue.

Via the SVD, \(X = UDV^\top\), the right singular vectors \(V\) are exactly those eigenvectors and \(\Sigma\)'s eigenvalues are \(d_j^2/n\). The Eckart-Young theorem says truncating to the top \(r\) singular values gives the best rank-\(r\) approximation of \(X\) in both Frobenius and spectral norm, which is the reconstruction statement.

The practical consequence is that you should compute PCA through the SVD of \(X\), never through the eigendecomposition of \(X^\top X\). Forming the covariance matrix squares the condition number and costs \(O(np^2)\) memory in \(p^2\) entries; a truncated SVD gets the top \(r\) components in roughly \(O(npr)\) without ever materialising \(\Sigma\).

Centring, scaling, and what changes

Centring is mandatory. Skip it and the first component points toward the mean of the data rather than toward its dominant direction of variation, which is a different and usually uninteresting quantity.

Scaling is a modelling decision. PCA on the covariance matrix lets high-variance features dominate, which is right when features share units and their scales are meaningful (pixel intensities, all-in-dollars financials). PCA on the correlation matrix, meaning standardise first, treats every feature as equally important, which is right when units are arbitrary. The two give genuinely different components, and reporting "PCA" without saying which was done leaves the result ambiguous.

Whitening goes one step further: after projecting, divide each component by its singular value so the transformed data have identity covariance. This equalises the contribution of every direction, which helps optimisers that are sensitive to conditioning. It also amplifies the low-variance directions, which are exactly the ones dominated by noise, so whitening a full-rank decomposition typically amplifies noise. Truncate first, then whiten.

When it breaks

Explained variance is not information. The top components capture variance, and variance is not relevance. In a classification problem the discriminative direction can lie in a low-variance component, and dropping it because it "explains 2%" discards the signal. Linear discriminant analysis optimises class separation instead and gives a different subspace for exactly this reason.

It is linear, and it only ever finds a subspace. Data on a curved manifold (a spiral, a torus) needs many linear components to approximate a structure with few intrinsic dimensions. Kernel PCA, autoencoders and neighbour embeddings exist for this; PCA's fixed inductive bias is a global linear subspace and nothing else.

Components are not features with meaning. A component is a signed linear combination of all inputs, and its sign is arbitrary: any implementation may return \(-w\) instead of \(w\). Interpreting loadings as a latent construct is a factor-analysis claim requiring rotation and domain argument, not something PCA delivers on its own.

Outliers dominate. Because the objective is squared error, a single extreme point can rotate the leading component substantially. Robust PCA variants exist; plain PCA has no defence, and inspecting the top components for outlier-driven directions is a routine sanity check.

The projection is fitted, so it leaks. Fitting PCA on the full dataset before splitting lets test-set structure inform the components. It has to be fitted on training data and applied to validation and test, like any other learned transform.