P-Values, Multiplicity and the Garden of Forking Paths

A model evaluation reports that a new checkpoint beats the incumbent on 3 of 24 benchmark suites at \(p < 0.05\). If the checkpoints were identical, the expected number of suites reaching \(p < 0.05\) by chance alone is \(24 \times 0.05 = 1.2\), and the probability of seeing at least one is \(1 - 0.95^{24} = 71\%\). Three is more than chance would usually give, but it is not the result the headline implies, and no correction was applied.

A p-value is \(P(\text{data at least this extreme} \mid H_0)\). It is a statement about data under an assumed null, not about the probability that the null is true, and not about the size of an effect. Every widespread misuse follows from swapping one of those.

What the number does and does not say

Three properties are worth holding precisely.

Under a true null with a continuous test statistic, the p-value is uniform on \([0,1]\). That is why \(\alpha = 0.05\) produces a 5% false-positive rate, and it is the whole mechanism behind multiplicity problems: uniform draws hit the tail regularly if you take enough of them.

The p-value confounds effect size with sample size. A trivial 0.02% difference reaches \(p < 0.001\) with enough traffic; a decisive 8% difference fails to reach \(p < 0.05\) on a small segment. Reporting an interval on the effect keeps these separable; reporting only significance throws the distinction away.

A non-significant result is not evidence of no effect. It is a failure to distinguish the observed data from the null, which can equally mean the study lacked power. Distinguishing "no effect" from "no information" requires either an equivalence test against a pre-specified margin or a Bayesian posterior, and the choice should be made before the data arrive.

Two different things you might want to control

Once there are \(m\) tests, "the error rate" becomes ambiguous, and the two standard answers differ in what they protect.

Family-wise error rate (FWER) is \(P(\text{at least one false positive among all } m)\). Bonferroni tests each hypothesis at \(\alpha/m\) and controls FWER under any dependence structure, which is why it survives despite being conservative. Holm's step-down procedure controls the same quantity uniformly more powerfully and costs nothing extra to run, so plain Bonferroni is almost always the wrong choice when Holm is available.

False discovery rate (FDR) is the expected proportion of rejections that are false. The Benjamini-Hochberg procedure sorts p-values ascending, finds the largest \(k\) with \(p_{(k)} \leq \frac{k}{m}\alpha\), and rejects the first \(k\) (Benjamini and Hochberg, 1995, JRSS-B 57(1), 289-300).

The choice is a decision-theoretic one, not a statistical fashion. Controlling FWER is right when a single false positive is costly: a safety gate, a regulatory claim, one confirmatory endpoint. Controlling FDR is right when the output is a ranked list to investigate further and a modest fraction of duds is tolerable: feature screening, ablation sweeps, genome-wide scans. Applying FWER to a 10,000-hypothesis screen guarantees you find nothing; applying FDR to a single go/no-go decision is not a correction at all.

The forking paths problem

The harder version of multiplicity does not involve multiple tests at all. Gelman and Loken's point is that the analysis you would have run on different data counts against you even if you only ran one analysis on the data you got (Gelman and Loken, 2013, The Garden of Forking Paths). Choosing the outlier rule after seeing the outliers, picking the metric that moved, splitting by device because the aggregate was flat: each decision is defensible on its own and each one is a hidden test.

This is why pre-registration and analysis plans do real work. They do not make the analysis smarter; they fix the number of tests at a value you can honestly count. The engineering equivalent is a held-out set that is genuinely touched once, and a written statement of the primary metric before the run starts.

When it breaks

Dependence between tests. Benjamini-Hochberg controls FDR exactly under independence and under positive regression dependence, a condition that covers many realistic correlated-metric cases but not all. Under arbitrary dependence you need the Benjamini-Yekutieli variant, which divides by \(\sum_{i=1}^{m} 1/i\) and is substantially more conservative. Correlated benchmark suites that share underlying data sit in the uncomfortable middle.

Sequential looking. Fixed-sample p-values assume the sample size was fixed in advance. Continuous monitoring inflates the type I error far beyond the nominal level, and the inflation grows with the number of looks. This has its own machinery (group-sequential boundaries, always-valid p-values) and is not fixed by any multiplicity correction.

Discreteness. With rare-event counts, the p-value is not uniform under the null; it takes a small number of achievable values, and the achieved type I error can be far below the nominal \(\alpha\). Corrections designed for continuous statistics become even more conservative here.

Power collapses quietly. Every correction trades false positives for false negatives. Bonferroni across 200 metrics at \(\alpha = 0.05\) tests each at \(0.00025\), which for a typical effect size means the study is now powered to detect approximately nothing. The honest response is to designate one primary metric, not to correct 200 of them into uselessness.