Likelihood Ratio, Wald and Score Tests

Fifty loan applications from a new channel produce 2 defaults. The historical default rate is 20%. Three textbook tests of \(p = 0.2\), all asymptotically \(\chi^2_1\) under the null, return statistics of 33.3, 11.1 and 8.0. Each rejects at the 5% level, but the implied evidence differs by a factor of four, and with slightly different counts one of them would reject while another would not. These are the likelihood ratio, Wald and score tests, sometimes called the trinity, and their disagreement is a direct readout of how far the log-likelihood is from a parabola.

Three measurements of one curve

Let \(\ell(\theta)\) be the log-likelihood for a parameter \(\theta \in \mathbb{R}^p\), with maximum likelihood estimate \(\hat\theta\) (see maximum-likelihood estimation), score \(U(\theta) = \nabla \ell(\theta)\) and Fisher information \(I(\theta)\). For the simple null \(H_0: \theta = \theta_0\):

Geometrically, LR measures the vertical drop in log-likelihood from the peak to the null. Wald measures the horizontal distance from \(\hat\theta\) to \(\theta_0\), scaled by curvature at the peak. Score measures the slope at the null, scaled by curvature at the null. If \(\ell\) is exactly quadratic, the three are numerically identical and exactly \(\chi^2\) at every sample size (Buse, 1982, The Likelihood Ratio, Wald, and Lagrange Multiplier Tests: An Expository Note, The American Statistician 36(3)).

Under regularity conditions and a true null, each converges to \(\chi^2_r\), where \(r\) is the number of restrictions. For LR this is Wilks' theorem (Wilks, 1938, The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses, Ann. Math. Statist. 9(1)). The score test is due to Rao (Rao, 1948, Proc. Cambridge Phil. Soc. 44(1)) and is known in econometrics as the Lagrange multiplier test. For composite nulls with nuisance parameters, LR compares the constrained and unconstrained maxima, Wald uses only the unconstrained fit, and score uses only the constrained one.

The binomial case, computed

With \(x = 2\) successes in \(n = 50\) trials, \(\hat p = 0.04\) and \(p_0 = 0.2\). The binomial information per trial is \(1/(p(1-p))\).

• Wald evaluates the variance at \(\hat p\): \(W = (0.04 - 0.2)^2 / (0.04 \times 0.96 / 50) = 0.0256 / 0.000768 \approx 33.3\).
• Score evaluates it at \(p_0\): \(S = 0.0256 / (0.2 \times 0.8 / 50) = 0.0256/0.0032 = 8.0\).
• LR uses the full curve: \(2[2\ln(0.04/0.2) + 48\ln(0.96/0.8)] \approx 11.1\).

The Wald statistic is inflated because the estimated variance at \(\hat p = 0.04\) is tiny; near the boundary, the curvature at the peak badly overstates precision. This is the same pathology that makes the Wald interval for a proportion undercover, and inverting the score test instead yields the Wilson interval discussed in confidence intervals and coverage.

Choosing among them

Cost. Wald needs only the full model, so every regression table's \(z\) column is a Wald test, free with the fit. Score needs only the restricted model, which makes it the natural tool for asking whether adding a term would help without fitting the larger model; Pearson's \(\chi^2\) test for a contingency table is a score test. LR needs both fits.

Invariance. LR is invariant to reparameterisation: testing \(p = 0.2\) or \(\operatorname{logit}(p) = \operatorname{logit}(0.2)\) gives the same statistic. Wald is not. The same hypothesis tested on the probability scale and the log-odds scale gives different \(W\) and can give different decisions, which is hard to defend.

Ordering. In the linear regression model with normal errors the three satisfy \(W \ge \text{LR} \ge S\) in every sample, so Wald rejects most often and score least (Berndt & Savin, 1977, Conflict Among Criteria for Testing Hypotheses in the Multivariate Linear Regression Model, Econometrica 45(5)). Asymptotic equivalence does not imply finite-sample agreement, and the ordering means a researcher free to choose the test after seeing it can choose the result.

The prevailing advice among statisticians is to prefer LR when both fits are affordable. Econometric practice leans harder on Wald because heteroskedasticity-robust and cluster-robust standard errors plug directly into it, and there is no equally convenient robust LR statistic. The disagreement is less about theory than about which failure you would rather have.

When it breaks

The Hauck–Donner effect. In logistic regression, as a coefficient's true magnitude grows, its estimated standard error grows faster, so the Wald statistic eventually decreases toward zero and power collapses to the significance level (Hauck & Donner, 1977, Wald's Test as Applied to Hypotheses in Logit Analysis, JASA 72(360)). A highly predictive rare-event feature can show a non-significant \(z\). Under quasi-separation the MLE diverges and the Wald test is meaningless; LR remains usable.

Boundaries break the \(\chi^2\) reference. Testing whether a variance component is zero puts \(\theta_0\) on the edge of the parameter space. The LR statistic's null distribution is then a mixture, commonly half a point mass at zero and half \(\chi^2_1\) for a single variance, and using \(\chi^2_1\) makes the test conservative. Mixture-model tests for the number of components fail regularity more severely still.

Non-nested models are out of scope. All three test restrictions on one model. Comparing a gamma regression with a log-normal one is not a restriction, and Wilks' theorem says nothing about it.

Misspecification changes the information. If the model is wrong, \(I(\theta)\) no longer equals the variance of the score, and all three statistics lose their \(\chi^2\) calibration. Sandwich variance estimates repair Wald and score cheaply; LR has no equally simple fix.