Ridge, Lasso and Elastic Net

A genomics model has 20,000 features and 300 samples. Ordinary least squares has infinitely many exact solutions, all of them useless. Add a penalty term and the problem becomes well-posed, uniquely solvable, and predictive. Which penalty you add decides what kind of answer you get: one that spreads signal across correlated features, or one that picks a few and zeroes the rest.

Both start from the same template, a loss plus a penalty on coefficient size:

with \(\Omega(\beta) = \|\beta\|_2^2\) for ridge and \(\|\beta\|_1\) for lasso. The intercept is never penalised, and features must be standardised first, since a penalty on raw coefficients means "penalise features measured in small units more", which is not a modelling decision anyone intends.

Ridge: shrinkage in the singular value basis

Ridge has a closed form, \(\hat{\beta}_{\text{ridge}} = (X^\top X + \lambda I)^{-1} X^\top y\), and adding \(\lambda I\) guarantees invertibility even when \(X^\top X\) is singular. The clearest view is through the SVD \(X = U D V^\top\). Ridge shrinks the component along the \(j\)-th right singular vector by the factor

Directions with large singular values, meaning high variance in the data, are barely touched. Directions with small singular values, the ill-conditioned ones where OLS estimates explode, are shrunk hard. Ridge is not a blunt instrument; it is a filter that suppresses exactly the directions the data do not constrain. Effective degrees of freedom, \(\sum_j d_j^2/(d_j^2 + \lambda)\), drops smoothly from \(p\) to 0 as \(\lambda\) grows.

Lasso: why the corners produce zeros

The lasso has no closed form and produces exact zeros, and the two facts have the same cause. Viewed as a constrained problem, minimise the residual sum of squares subject to \(\|\beta\|_1 \leq t\). The \(\ell_1\) ball is a cross-polytope: a diamond in two dimensions, with vertices on the axes. The elliptical contours of the squared-error loss expanding outward from the OLS solution touch this ball at a corner with high probability, and a corner is a point where some coordinates are exactly zero. The \(\ell_2\) ball is round, has no corners, and so touches at a generic point where nothing is zero.

Algebraically, in the orthonormal-design case, lasso is soft thresholding: \(\hat{\beta}_j = \mathrm{sign}(\beta_j^{\text{OLS}})(|\beta_j^{\text{OLS}}| - \lambda)_+\). Coefficients below \(\lambda\) are set to zero, and those above are pulled toward zero by exactly \(\lambda\). That second part matters: lasso's surviving coefficients are biased toward zero, which is why relaxed lasso and debiasing steps exist (Tibshirani, 1996, Regression Shrinkage and Selection via the Lasso, JRSS-B 58(1)).

Elastic net and the correlated-feature problem

Give lasso ten nearly identical features carrying the same signal and it selects roughly one, essentially arbitrarily; which one it picks can flip with a small perturbation of the data. Ridge instead spreads the coefficient across all ten, each getting about a tenth. Neither is universally right, and elastic net combines them:

The \(\ell_2\) component induces a grouping effect, so correlated features enter or leave together, while the \(\ell_1\) component still produces sparsity (Zou and Hastie, 2005, JRSS-B 67(2), 301-320). It also removes lasso's structural limit of selecting at most \(n\) features when \(p > n\).

When it breaks

Lasso's selection is unstable, and stability is not accuracy. A bootstrap over the same dataset can yield substantially different selected sets. If the deliverable is "which features matter", report selection frequency across resamples rather than one run's support. Prediction can be excellent while the selected set is nearly arbitrary.

Post-selection inference is invalid. Fitting OLS on the lasso-selected features and reporting the usual p-values ignores that selection used the same data. The intervals undercover, sometimes badly. Valid procedures exist (selective inference, the debiased lasso) and none of them is the naive two-step.

Cross-validation picks \(\lambda\) for prediction, not for recovering the truth. The \(\lambda\) minimising CV error is typically smaller than the one that would recover the true support, so the CV-optimal model tends to include extra features. The one-standard-error rule, choosing the largest \(\lambda\) within one standard error of the minimum, is the usual pragmatic correction.

Regularisation does not fix confounding or leakage. It controls variance. A leaked feature with genuine predictive power survives any penalty happily, and a confounded coefficient stays confounded after shrinkage. Penalties are a variance instrument, not a causal one.