Twenty Tests, One False Discovery: Multiple Testing From Bonferroni to the False Discovery Rate

The subject weighed 3.8 lbs, measured about 18 inches, and "was not alive at the time of scanning." Craig Bennett and colleagues scanned a mature Atlantic salmon while showing it 15 photographs of people in social situations. Out of 8,064 voxels, 16 were significant at an uncorrected \(p < 0.001\), clustered in the fish's brain cavity. Rerun with control of either the false discovery rate or the family-wise error rate, there were no active voxels, "even at relaxed statistical thresholds (p = 0.25)" (Bennett et al., 2009, NeuroImage 47 S125).

The joke had a payload: in 2008 only 74% of NeuroImage fMRI papers (193 of 260) applied any correction (Bennett et al., 2010, JSUR 1(1)). An eval harness scoring 40 benchmark slices, or a platform running 300 experiments with 12 metrics each, is a salmon scanner until someone decides what error rate the family should have.

Why this matters: Every ML and product workflow tests many hypotheses and reports the winners. Controlling the chance of any false positive (FWER) or the fraction of false discoveries (FDR) decides whether a screen finds nothing or finds noise, and online and adaptive settings (A/B platforms, a reused holdout) quietly break the classical procedures.

TL;DR

• Twenty independent null tests at \(\alpha = 0.05\) yield at least one false positive with probability \(1 - 0.95^{20} = 64\%\).
• Holm (1979) controls the family-wise error rate under any dependence and dominates Bonferroni at no cost.
• Under independence the Benjamini-Hochberg step-up rule gives \(\text{FDR} = \pi_0 q\) exactly, and the proof fits in five lines.
• In a 10,000-test simulation with 10% real effects, Holm recovered 7.9% of them and BH 60.7%, at a realised FDR of 4.4%.
• FDR is an average. With correlation 0.5 between tests, BH's false discovery proportion averaged 3.9% but exceeded 20% in 6.1% of runs.
• Storey's q-values estimate the null fraction and regain power: 160 genes at \(q \le 0.05\) on data where BH called 88.
• Online rules (alpha-investing, LORD++) treat alpha as wealth that discoveries replenish; the reusable holdout brings the same budget logic to model selection.

At a Glance

flowchart LR
    A["m hypotheses, one p-value each"] --> B{"Cost of one false positive?"}
    B -->|"Catastrophic"| C["FWER: Holm"]
    B -->|"Tolerable in a list"| D["FDR: BH, BY, Storey"]
    B -->|"Need per-item belief"| E["Local fdr"]
    A --> F{"Tests arrive over time?"}
    F -->|"Yes"| G["Online FDR: LORD++"]
    F -->|"Reusing one holdout"| H["Thresholdout"]
    C --> I["Few, reliable discoveries"]
    D --> J["Many discoveries, bounded dud rate"]
    G --> J
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    class A blue
    class B,F amber
    class C,D,E,G,H purple
    class I,J teal

From a Union Bound to a Rate of Discoveries

For decades, correcting for multiple comparisons meant bounding the probability of any false rejection with the union bound named for Carlo Emilio Bonferroni: run each of \(m\) tests at \(\alpha/m\) and \(P(\text{any false rejection}) \le m \cdot \alpha/m = \alpha\), whatever the dependence. In 1979 Sture Holm showed that much of the price was unnecessary: test the smallest p-value at \(\alpha/m\), the next at \(\alpha/(m-1)\), and stop at the first failure (Holm, 1979, Scand. J. Statist. 6(2)). Same guarantee, uniformly more power. In 1986 John Simes proposed critical values \(k\alpha/m\) for a global test (Simes, 1986, Biometrika 73(3)); they would soon be put to another use.

The shift was a change of question. Benjamini and Hochberg observed that investigators often need the list of rejections to be mostly right, not every item. They defined the false discovery rate as the expected proportion of false rejections among rejections and showed that a step-up rule on Simes' values controls it (Benjamini and Hochberg, 1995, JRSS-B 57(1)). Microarrays soon arrived with thousands of genes per chip, where FWER control returns almost nothing.

timeline
    title Multiple testing, from union bounds to online discovery
    1979 : Holm publishes the sequentially rejective FWER procedure
    1995 : Benjamini and Hochberg define the false discovery rate
    2001 : Benjamini and Yekutieli prove BH under positive dependence
         : Efron and colleagues frame FDR as empirical Bayes
    2002 : Storey estimates the null fraction and defines q-values
    2005 : Ioannidis argues most published findings are false
    2008 : Foster and Stine publish alpha-investing
    2009 : The dead salmon poster at Human Brain Mapping
    2015 : Dwork et al. publish the reusable holdout in Science
         : Javanmard and Montanari propose LORD for online FDR

Two threads follow. Efron, Tibshirani, Storey and Tusher's empirical Bayes analysis (Efron et al., 2001, JASA 96) recast FDR as a posterior probability. Foster and Stine's alpha-investing (2008, JRSS-B 70(2)) asked how to decide before the next p-value exists, the situation of every experimentation platform.

How False Discovery Control Actually Works

Two error rates as random variables

Run \(m\) tests, \(m_0\) with a true null. A procedure rejects \(R\) (observed), of which \(V\) (unobserved) are false. Then

Because \(\text{FDP} \le \mathbf{1}\{V \ge 1\}\) pointwise, \(\text{FDR} \le \text{FWER}\): FDR is the weaker demand. When every null is true, any rejection is false, \(\text{FDP} = \mathbf{1}\{V \ge 1\}\), and the two coincide. So FDR control still protects against finding something in pure noise, the salmon's failure, but never protects a single discovery: it is a property of the list.

Holm: spending alpha against the worst case

Sort \(p_{(1)} \le \dots \le p_{(m)}\). Holm rejects up to the first \(k\) with \(p_{(k)} > \alpha/(m-k+1)\). The proof counts. With probability at least \(1-\alpha\) (union bound), every true null has \(p > \alpha/m_0\). On that event, suppose the first false rejection happens at step \(j\). Every earlier rejection was a false null, so \(j \le m - m_0 + 1\) and the threshold \(\alpha/(m-j+1) \le \alpha/m_0\), which no true null can pass. Contradiction; no dependence assumption used.

The cost: at \(m = 10{,}000\) the first test runs at \(5 \times 10^{-6}\), a z-score of about 4.4, so a three-standard-error effect clears it only if noise adds another 1.4.

Benjamini-Hochberg: a line through the origin

BH at level \(q\) finds

and rejects \(H_{(1)}, \dots, H_{(k^\ast)}\). Plot sorted p-values against rank, draw a line of slope \(q/m\) through the origin, and reject up to the last point under it. Holm steps down and stops at the first failure; BH steps up, so a p-value above its own threshold is still rejected if a larger one passes.

The line has a plug-in reading. Rejecting every \(p_i \le t\) produces about \(m_0 t \le m t\) null rejections in expectation, and \(R(t) = \#\{p_i \le t\}\) rejections in total, so

is a conservative FDP estimate. BH is exactly "take the largest \(t\) with \(\widehat{\text{FDR}}(t) \le q\)": at \(t = p_{(k)}\), \(R = k\) and the condition is \(m p_{(k)}/k \le q\).

Why BH works: the proof sketch

Assume null p-values are uniform and independent of each other and of the non-nulls. Write the FDR as a sum over the set of true nulls \(I_0\):

Fix a null \(i\) and let \(R^{(i)}\) be the number of rejections BH would make if \(p_i\) were replaced by 0; it depends only on the other \(m-1\) p-values. The key lemma: \(H_i\) is rejected with \(R = k\) exactly when \(p_i \le kq/m\) and \(R^{(i)} = k\). Then

using independence to factor the joint probability, uniformity for \(P(p_i \le kq/m) = kq/m\), and the fact that \(R^{(i)}\) takes exactly one value. Summing over \(m_0\) nulls,

BH is not just bounded by \(q\); under independence it spends exactly \(\pi_0 q\) and wastes the rest. The \(1/k\) in the FDP cancels the \(k\) in the threshold, which is why thresholds grow linearly in rank, and independence was used only in the factorisation, which says exactly what to worry about under correlation.

Dependence: PRDS and the Benjamini-Yekutieli price

The factorisation becomes an inequality when p-values are positively regression dependent on the nulls (PRDS), which covers one-sided tests on positively correlated Gaussian statistics. Under arbitrary dependence, BH must run at \(q/c(m)\) with

(Benjamini and Yekutieli, 2001, Ann. Statist. 29(4)). The constant is 2.93 at \(m = 10\) and 9.79 at \(m = 10{,}000\). In the simulation below BY recovered 25.9% of true effects against BH's 60.7%, with a realised FDR of 0.4%.

Storey: estimating what BH wastes

If \(\text{FDR} = \pi_0 q\), running BH at \(q/\pi_0\) would hit \(q\) exactly. Storey estimated \(\pi_0\) from the flat right side of the p-value histogram, where nearly every p-value is null (Storey, 2002, JRSS-B 64(3)):

About \(m_0(1-\lambda)\) nulls land above \(\lambda\), so this recovers \(m_0/m\). The q-value of a test is the smallest estimated FDR at which it would be called significant, the FDR analogue of a p-value. On the Hedenfalk breast-cancer data (3,170 genes, \(\hat{\pi}_0 = 0.67\) at \(\lambda = 0.5\)), \(q \le 0.05\) called 160 genes, of which "≈8 ... are expected to be false positives," while BH found 88 (Storey and Tibshirani, 2003, PNAS 100(16)). When \(\pi_0\) is near 1, as on most A/B platforms, the gain disappears.

[IMAGE: Histogram of 3,170 p-values with a flat right half. A dashed horizontal line at the flat height marks pi0 = 0.67; the area above it near zero is shaded emerald as estimated non-nulls. Caption: "Storey reads the null fraction off the flat right half of the histogram and hands the unused budget back to BH."]

Efron: the local false discovery rate

Tail-area FDR describes a list; the two-group model describes an item. If \(f(z) = \pi_0 f_0(z) + \pi_1 f_1(z)\), Bayes' rule gives

and the tail-area \(\text{Fdr}(z) = \pi_0 F_0(z)/F(z)\) is the average of \(\text{fdr}\) over the tail. Because the average includes far more convincing extreme cases, the last item on a 5% FDR list can have a local fdr several times larger than 5%. Efron proposed \(\text{fdr} \le 0.2\) as a reporting threshold; on a prostate study of 6,033 genes, 51 met it (Efron, 2007, Ann. Statist. 35(4)). The same paper shows the theoretical null can be wrong: on an HIV microarray (7,680 genes, 8 arrays) the central z-histogram is "substantially too narrow" for \(N(0,1)\), so Efron estimates an empirical null from the data.

Online FDR: alpha as a currency

BH needs all \(m\) p-values before deciding anything; a platform must decide as each test ends. Foster and Stine's alpha-investing starts with wealth \(W(0) \le \alpha\), pays \(\alpha_j/(1-\alpha_j)\) when test \(j\) fails to reject and earns \(\omega\) when it rejects, controlling a modified rate, mFDR, while wealth stays nonnegative. Javanmard and Montanari's LORD controls FDR itself: pick \(\gamma_1, \gamma_2, \dots\) summing to \(\alpha\), test at \(\gamma_i\), and restart the sequence after each discovery (Javanmard and Montanari, 2018, Ann. Statist. 46(2)). Ramdas and colleagues' LORD++ lets every past discovery keep paying in:

with \(\tau_j\) the time of the \(j\)-th rejection and defaults \(W_0 = \alpha/2\), \(\gamma_j = 0.0722 \log(j \vee 2) / (j e^{\sqrt{\log j}})\) (Ramdas et al., 2017, arXiv:1710.00499). Discoveries buy future power; droughts starve the procedure.

The reusable holdout: multiplicity without p-values

Model selection on a fixed test set is multiple testing with adaptively chosen hypotheses. Dwork and colleagues built a null world: \(n = 10{,}000\) points, \(d = 10{,}000\) features, labels independent of the data, so nothing beats 50%. An analyst picks features correlated with the label on training data, keeps those whose sign agrees on the holdout, and builds a classifier. "Reusing a standard holdout results in reported accuracy of over 63% for k = 500 on both the training set and the holdout set" (Dwork et al., 2015, arXiv:1506.02629; Science 349(6248)). Their Thresholdout returns the training mean unless it differs from the holdout mean by more than a noisy threshold, only then revealing a noised holdout value. Most queries leak one bit, and differential privacy turns that into a guarantee.

Seeing It in Motion

Which rate to control is a decision about consequences:

flowchart TB
    S["A family of m tests"] --> Q1{"Any single false claim unacceptable?"}
    Q1 -->|"Yes, a safety gate"| H["Holm at alpha"]
    Q1 -->|"No, output is a shortlist"| Q2{"Dependence structure?"}
    Q2 -->|"Independent or PRDS"| Q3{"Many true signals?"}
    Q2 -->|"Arbitrary"| BY["Benjamini-Yekutieli"]
    Q3 -->|"Yes, pi0 well below 1"| ST["Storey q-values"]
    Q3 -->|"No, pi0 near 1"| BH["Plain BH"]
    ST --> L["Report local fdr at the boundary"]
    BH --> L
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    class S blue
    class Q1,Q2,Q3 amber
    class H,BY,ST,BH purple
    class L teal

The online case is a stream. Below, LORD++ at \(\alpha = 0.05\) with the default \(W_0\) and \(\gamma_j\) (\(\gamma_1 \approx 0.050\), \(\gamma_2 \approx 0.011\)) meets four experiments.

sequenceDiagram
    participant E as Experiment stream
    participant L as LORD++ controller
    participant W as Alpha-wealth ledger
    participant D as Ship decision
    E->>L: Test 1 finishes, p = 0.0004
    L->>W: Level set in advance, 0.00125
    W-->>D: Reject, discovery at t = 1
    E->>L: Test 2 finishes, p = 0.03
    L->>W: Level 0.00152, boosted by discovery 1
    W-->>D: Fail to reject, keep control
    E->>L: Test 3 finishes, p = 0.0003
    L->>W: Level 0.00050, decaying since t = 1
    W-->>D: Reject, discovery at t = 3
    E->>L: Test 4 finishes
    L->>W: Level 0.00293, two discoveries pay in
    Note over L,W: Without discoveries every level decays toward zero

The p-values are invented; the levels are what the formula computes. Test 3 faced a bar six times stricter than test 4, purely because of where it fell in the stream, and every level is fixed before its p-value is seen.

[IMAGE: Line chart of LORD++ levels alpha_t over 200 sequential experiments on a log y-axis. Ticks mark discoveries at t = 12, 13, 60 and 140; after each, the level jumps then decays along the gamma sequence. A shaded band from t = 60 to 140 marks a drought where levels fall below 0.0005. Caption: "Online FDR spends alpha in proportion to recent success; a platform that mostly ships nulls pays for it in power."]

[IMAGE: Two panels from the reusable holdout experiment. Left: accuracy on training, standard holdout and fresh data versus number of selected variables (0 to 500); training and holdout climb above 0.63 while fresh stays at 0.50. Right: with Thresholdout, reported holdout accuracy stays near 0.50. Caption: "Adaptive reuse overfits the holdout itself; answering most queries with a single bit prevents it (Dwork et al., 2015)."]

By the Numbers

Sources: Bennett et al. (2009, 2010); Storey and Tibshirani (2003); Ioannidis (2005, Table 4); Colquhoun (2014); Jager and Leek (2014). The simulation rows are this author's own runs (200 and 2,000 replications, one-sided z-tests, \(q = 0.05\)), included to illustrate mechanism, not as published results.

The Ioannidis, Colquhoun and Jager-Leek rows are one quantity seen three ways, and they disagree loudly. Ioannidis's \(\text{PPV} = (1-\beta)R/(R - \beta R + \alpha)\) is one minus a Bayesian FDR with prior odds \(R\); Colquhoun's 36% is the same arithmetic at 10% prevalence (\(45/125\)). Jager and Leek fitted a two-group mixture to reported p-values and concluded that "most published medical research is true" (arXiv:1301.3718). The gap is not arithmetic; it is whether reported p-values are an unselected sample, which publication bias exists to violate.

A Concrete Example

Ten metrics from one product experiment, sorted. Target \(q = \alpha = 0.05\), \(m = 10\).

1. Uncorrected and Bonferroni. Eight metrics have \(p < 0.05\); at \(0.005\) only ranks 1 and 2 pass.
2. Holm. \(0.0011 \le 0.005\), \(0.0032 \le 0.00556\), \(0.0060 \le 0.00625\); then \(0.0215 > 0.00714\) and Holm stops. Three rejections, one more than Bonferroni for free.
3. BH, from the top. \(0.634 > 0.050\), \(0.213 > 0.045\), \(0.049 > 0.040\), \(0.041 > 0.035\), \(0.0392 > 0.030\), then \(0.0238 \le 0.025\). So \(k^\ast = 5\); reject ranks 1 to 5.
4. The step-up surprise. Rank 4 failed its own threshold (\(0.0215 > 0.020\)) and is rejected anyway. The effective cutoff is \(t = 0.0238\), where the plug-in estimate is \(m t / R = 10 \times 0.0238 / 5 = 0.0476 \le 0.05\).
5. Adjusted p-values. Take \(m p_{(k)}/k\) with a running minimum from the bottom: rank 4's \(0.05375\) becomes \(\min(0.05375, 0.0476) = 0.0476\).
6. BY. \(c(10) = 2.929\), so thresholds are \(k \times 0.001707\); rank 3 at 0.0060 misses 0.0051. Two rejections.
7. Storey, and why not here. Only 0.634 exceeds \(\lambda = 0.5\), so \(\hat\pi_0 = 1/(10 \times 0.5) = 0.2\); adaptive BH at \(0.25\) passes rank 9 (\(0.213 \le 0.225\)) and declares nine of ten real. With ten tests, \(\pi_0\) estimation is noise.

Two metrics survive a family-wise guarantee, five a 5% FDR list, eight the dashboard. Ranks 4 and 5 sit on the line and carry most of the list's risk.

[IMAGE: The worked example plotted: ten sorted p-values against rank, the BH line y = 0.005k, the flat Bonferroni line at 0.005 and Holm's staircase. Rank 4 circled in amber just above the BH line, rank 5 circled in green just below it,. Caption: "Step-up: the last crossing wins, so rank 4 is rejected despite failing its own threshold."]

Where It Breaks

The family is whatever you say it is

Nothing in the mathematics says what \(m\) is. Correct across the 12 metrics of one experiment and the guarantee covers that experiment; correct each of 300 experiments separately and nothing bounds the error rate of the quarter's shipped wins. Choosing the family after seeing the results is the forking-paths problem again; the only protection is naming it in advance.

FDR is a mean, and the variance is the problem

Under independence the FDP concentrates near \(\pi_0 q\). Under correlation it does not. In this post's simulation (1,000 tests, 100 real effects, equicorrelation 0.5), BH's mean FDP was 3.9%, yet 69% of runs had FDP zero and 6.1% had FDP above 20%, because a shared noise component occasionally drags many nulls into the tail together. When a bad list is costly, target the false discovery exceedance \(P(\text{FDP} > \gamma)\), which Javanmard and Montanari also treat.

p-values that were never uniform

Every guarantee assumes null p-values are uniform or conservative. Discrete tests (rare-event counts, Fisher's exact test) give lumpy, conservative p-values and lose power. Wrong variances under clustered randomisation, or a mis-specified null as in Efron's HIV data, give anticonservative ones, and no correction rescues the list. Checking the right half of the p-value histogram for flatness is the cheapest diagnostic available.

Alpha-death and queue-jumping online

LORD's levels decay between discoveries. A platform whose early tests are all nulls reaches a state where no later test can reject, which Ramdas and colleagues call alpha-death. Resetting the ledger each quarter silently starts a new family, and letting a promising test jump the queue after a peek breaks the conditional validity the proofs rely on.

Selection inflates what survives

Controlling the decision error does nothing for the estimate. Discoveries that barely cleared a threshold were selected for favourable noise, so rank 5's reported lift is almost certainly too large. "How big?" needs shrinkage, replication or selective inference.

The holdout theory overshoots in practice

Adaptive reuse can overfit a holdout badly; on large benchmarks it mostly has not. Rebuilding ImageNet's test set cost models 11% to 14% accuracy, but gains on the old test set translated to larger gains on the new one, pointing to distribution shift rather than adaptivity (Recht et al., 2019). A meta-analysis of over one hundred Kaggle competitions found "little evidence of substantial overfitting" (Roelofs et al., 2019, NeurIPS). Worst-case theory and typical-case evidence disagree.

[IMAGE: Histogram of realised false discovery proportion across 2,000 simulated BH runs at correlation 0.5: a tall bar at FDP = 0, a long right tail, a dashed line at the 3.9% mean, the tail beyond 20% shaded rose. Caption: "FDR bounds the mean of this distribution; under correlation one run in sixteen is a disaster."]

Alternative Designs

Bonferroni is absent because Holm dominates it. Knockoffs handle correlated regression features by constructing fake variables that mimic the real ones' correlation but are known nulls, and use each feature-knockoff pair's symmetry to estimate the FDP (Barber and Candès, 2015, Ann. Statist. 43(5)). Hierarchical Bayesian models instead shrink noisy extremes toward a shared mean, at the price of a prior someone must defend.

How It Is Used in Practice

Genomics made FDR the default after Storey and Tibshirani showed what FWER control cost on real arrays.

Experimentation platforms now build it into the product. Optimizely's Stats Engine uses "a tiered version of the Benjamini-Hochberg procedure" across metrics and variations, correcting the primary metric independently of the rest (Optimizely documentation). Online FDR targets the sequence of experiments over months; Ramdas and colleagues list "multiple A/B or multi-armed bandit tests for internet commerce" among its applications. The burden is a ledger of every test, in committed order, with its level.

Model evaluation is the least disciplined setting. A release note claiming wins on 7 of 40 benchmark slices at \(p < 0.05\) should say what identical checkpoints would show: 2 expected under independence, with far more spread when slices share items. The cheap fix is BH over slices, a declared primary metric, and a confirmation run on untouched data for anything cited.

[IMAGE: Flow diagram of a model-evaluation harness: 40 benchmark slices each yield a p-value, a BH step at q = 0.10 produces a shortlist, and only shortlisted slices are re-run on a fresh held-out split. Caption: "Screen with FDR, confirm on untouched data."]

Insights Worth Remembering

1. FDR is a property of a list, never of an item. Boundary items on a 5% list are far likelier than 5% to be null. Act on one alone only after confirming it.
2. BH spends exactly \(\pi_0 q\). Adaptive methods reclaim the slack, and gain nothing where almost every idea is a null.
3. Correlation leaves the mean intact and wrecks the distribution. Averages protect institutions running many studies; not a single study.
4. Multiplicity correction does not fix effect sizes. Whatever survives a threshold is biased upward. Decisions and estimates need separate treatment.
5. Online FDR turns alpha into a currency. A platform that mostly ships nulls gets poorer. Putting likely wins first raises power, if the order is fixed before results arrive.
6. The fight over published science is a fight about priors. Ioannidis, Colquhoun and Jager-Leek use the same Bayes-rule arithmetic and differ on pre-study odds and on how reported p-values were selected.

Open Questions

Can FDR be defined meaningfully across an organisation's entire experiment history? Online methods handle one stream, but real platforms have many teams, overlapping metrics and asynchronous tests. Decaying-memory FDR (Ramdas et al., 2017) is one proposal; whether any global guarantee is operationally achievable is open.

What is the right procedure under the dependence of modern ML evaluation? Benchmark slices share items and models share pretraining data. BY is valid but weak; knockoffs and e-value methods are candidates. Whether BH's robustness transfers beyond Gaussian-like correlation is unmeasured.

When does adaptive test-set reuse actually hurt? Theory says the damage can be large; Kaggle and ImageNet evidence says it usually was not. Whether that survives a few large labs iterating on a handful of shared LLM benchmarks is open.

Should local fdr replace q-values as the reported quantity? It answers the question users actually ask about an item, but needs density estimation that is fragile at small \(m\) and in the extreme tails. Evidence that practitioners interpret it better is anecdotal.

Sources and Further Reading

1. Holm, S. (1979). "A Simple Sequentially Rejective Multiple Test Procedure." Scandinavian Journal of Statistics, 6(2), 65-70. JSTOR
2. Simes, R. J. (1986). "An improved Bonferroni procedure for multiple tests of significance." Biometrika, 73(3), 751-754. DOI
3. Benjamini, Y., & Hochberg, Y. (1995). "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing." JRSS-B, 57(1), 289-300. DOI
4. Benjamini, Y., & Yekutieli, D. (2001). "The control of the false discovery rate in multiple testing under dependency." Annals of Statistics, 29(4). DOI
5. Efron, B., Tibshirani, R., Storey, J. D., & Tusher, V. (2001). "Empirical Bayes Analysis of a Microarray Experiment." JASA, 96(456), 1151-1160. DOI
6. Storey, J. D. (2002). "A Direct Approach to False Discovery Rates." JRSS-B, 64(3), 479-498. DOI
7. Storey, J. D., & Tibshirani, R. (2003). "Statistical significance for genomewide studies." PNAS, 100(16), 9440-9445. DOI
8. Ioannidis, J. P. A. (2005). "Why Most Published Research Findings Are False." PLoS Medicine, 2(8), e124. DOI
9. Efron, B. (2007). "Size, power and false discovery rates." Annals of Statistics, 35(4). arXiv:0710.2245
10. Foster, D. P., & Stine, R. A. (2008). "α-Investing: a Procedure for Sequential Control of Expected False Discoveries." JRSS-B, 70(2), 429-444. DOI
11. Bennett, C. M., Baird, A. A., Miller, M. B., & Wolford, G. L. "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon." NeuroImage 47, S125 (2009) DOI; Journal of Serendipitous and Unexpected Results 1(1), 1-5 PDF
12. Jager, L. R., & Leek, J. T. (2014). "An estimate of the science-wise false discovery rate and application to the top medical literature." Biostatistics, 15(1), 1-12 DOI; Colquhoun, D. (2014). "An investigation of the false discovery rate and the misinterpretation of p-values." Royal Society Open Science, 1, 140216 DOI
13. Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015). "The reusable holdout: Preserving validity in adaptive data analysis." Science, 349(6248), 636-638 DOI; arXiv:1506.02629
14. Barber, R. F., & Candès, E. J. (2015). "Controlling the false discovery rate via knockoffs." Annals of Statistics, 43(5). DOI
15. Javanmard, A., & Montanari, A. (2018). "Online rules for control of false discovery rate and false discovery exceedance." Annals of Statistics, 46(2) DOI; Ramdas, A., Yang, F., Wainwright, M. J., & Jordan, M. I. (2017). "Online control of the false discovery rate with decaying memory." arXiv:1710.00499
16. Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). "Do ImageNet Classifiers Generalize to ImageNet?" arXiv:1902.10811; Roelofs, R., et al. (2019). "A Meta-Analysis of Overfitting in Machine Learning." NeurIPS. Proceedings

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.