Hypothesis Tests, Error Types and the Neyman–Pearson Lemma

A fraud team can flag every transaction and never miss a fraudster, or flag none and never inconvenience a customer. Every useful rule sits between those extremes, and the only interesting question is which rule gives the fewest misses for the number of false alarms you can tolerate. Neyman and Pearson posed hypothesis testing as exactly that question in 1933, and answered it for the simplest case with a result that still organises the whole subject (Neyman & Pearson, 1933, On the Problem of the Most Efficient Tests of Statistical Hypotheses, Phil. Trans. R. Soc. A 231).

A test is a decision rule with two error rates

Data \(X\) come from a density \(f_\theta\). The null hypothesis \(H_0: \theta \in \Theta_0\) is set against an alternative \(H_1: \theta \in \Theta_1\). A test is a function \(\phi(X) \in [0,1]\) giving the probability of rejecting \(H_0\); for most tests it is 0 or 1, determined by a rejection region \(R\).

Two errors are possible. A Type I error rejects a true null, and the test's size is \(\alpha = \sup_{\theta \in \Theta_0} P_\theta(X \in R)\). A Type II error fails to reject a false null, with probability \(\beta(\theta)\) at a given alternative. Power is \(1 - \beta(\theta)\), and it is a function of the alternative, not a single number.

For fixed data the two errors trade: shrinking \(R\) lowers \(\alpha\) and raises \(\beta\). The framework fixes \(\alpha\) in advance and asks for the most power it can buy, so the null should be the hypothesis whose false rejection costs more.

The lemma

Consider the simplest case, a simple null \(\theta_0\) against a simple alternative \(\theta_1\). Define the likelihood ratio

The Neyman–Pearson lemma states that the test rejecting when \(\Lambda(x) > k\), with \(k\) chosen so that \(P_{\theta_0}(\Lambda(X) > k) = \alpha\), has the greatest power among all tests of size at most \(\alpha\). When the data are discrete and no \(k\) hits \(\alpha\) exactly, the optimal test randomises on the boundary \(\Lambda(x) = k\).

The two-line proof explains the result. For any other test \(\phi'\) of size at most \(\alpha\), the integrand \((\phi(x) - \phi'(x))(f_{\theta_1}(x) - k f_{\theta_0}(x))\) is non-negative everywhere, because \(\phi = 1\) exactly where \(f_{\theta_1} > k f_{\theta_0}\). Integrating gives power\((\phi)\) \(-\) power\((\phi') \ge k(\alpha - \alpha') \ge 0\). The lemma is a knapsack argument: each point of the sample space costs null probability and earns alternative probability, and you fill the rejection region in order of the earnings-to-cost ratio.

Worked numbers

Let \(X_1,\dots,X_9 \sim \mathcal{N}(\mu, 1)\), testing \(\mu = 0\) against \(\mu = 1\). The log likelihood ratio is \(\sum_i x_i - 9/2\), increasing in \(\bar x\), so the optimal test rejects when \(\bar x > c\). Size 0.05 requires \(c = 1.645/\sqrt{9} \approx 0.548\). Power at \(\mu = 1\) is \(P(Z > (0.548 - 1)\times 3) = P(Z > -1.355) \approx 0.91\).

Notice that the rejection region did not depend on the alternative value 1, only on its direction. Any \(\mu_1 > 0\) gives the same test. That is the route from the lemma to a uniformly most powerful test for the one-sided composite alternative \(\mu > 0\): when the likelihood ratio is monotone in a statistic, the Karlin–Rubin theorem makes the one-sided threshold test UMP. For the two-sided alternative \(\mu \ne 0\) no UMP test exists, because the best test against \(\mu = 1\) and the best test against \(\mu = -1\) reject in opposite tails. Restricting to unbiased tests restores an optimum, and that restriction is a choice, not a consequence.

Two schools that the textbook merges

Textbooks teach Fisher's significance test and the Neyman–Pearson procedure as one method. Fisher treated the p-value as a graded measure of evidence against a single null, with no alternative and no Type II error. Neyman and Pearson specified \(\alpha\) before the data, required an alternative, and made only a binary decision whose long-run error rates were the entire justification. Lehmann argued that the two are complementary in practice and can be unified (Lehmann, 1993, The Fisher, Neyman–Pearson Theories of Testing Hypotheses: One Theory or Two?, JASA 88(424)); others hold that the hybrid, reporting an exact p-value and then comparing it to 0.05, inherits the weaknesses of both. The p-value side of this is covered in p-values, multiplicity and forking paths; the planning side in statistical power and the minimum detectable effect.

When it breaks

Real hypotheses are composite. The lemma is exact only for simple against simple. Nuisance parameters, such as an unknown variance, turn the null into a set, and optimality then depends on invariance or conditioning arguments that hold in some families and not others. The generalised likelihood ratio test is the workhorse replacement, and its optimality is asymptotic at best.

Fixed \(\alpha\) makes large samples reject everything. With \(n = 10^6\), a difference of 0.003 standard deviations is significant at 0.05. The test has done its job; the question it answered, whether the effect is exactly zero, was rarely the one that mattered. An effect-size interval answers the decision question the test does not.

Randomised tests are optimal and unused. Exact size on discrete data needs a coin flip at the boundary, which nobody reports. Never rejecting there instead gives actual size below nominal and loses power, badly for small binomial samples.

Error rates are pre-data guarantees. Stopping when the result looks good, or choosing the test after seeing the data, voids the \(\alpha\) the procedure advertises. The guarantee belongs to the whole protocol, and a protocol altered mid-stream has no known error rate.