Peeking and Sequential Testing

An experiment dashboard updates hourly. The team watches it, sees \(p = 0.04\) on day three, and ships. The stated false-positive rate was 5%. The actual false-positive rate, given that they would have kept watching had it not crossed, is substantially higher, and under continuous monitoring with an unbounded horizon the probability of crossing a fixed threshold at some point approaches 1 even when the true effect is exactly zero.

The p-value's guarantee is conditional on a sampling plan. Fixed-sample inference assumes \(n\) was fixed before data collection. Choosing \(n\) by looking at the data breaks that condition, and no correction applied at the end repairs it.

Why the inflation is large

Under the null, the test statistic follows a random walk. A fixed threshold is crossed with probability \(\alpha\) at any single pre-specified moment, but the probability that a random walk touches a fixed boundary at some point during a long run is much higher, and grows with the number of looks.

Johari, Pekelis and Walsh quantified this for realistic monitoring patterns and built the standard alternative (Johari, Pekelis and Walsh, 2015, Always Valid Inference, arXiv:1512.04922; expanded in Johari, Koomen, Pekelis and Walsh, 2017, Peeking at A/B Tests, KDD). The practical significance of their framing is that the problem is caused by the interface: giving experimenters a live significance readout and expecting them not to act on it is a process design that fails.

Three legitimate responses

Fix the horizon and do not look. Compute the sample size, run to it, analyse once. Statistically correct and organisationally fragile: someone will look, and a genuinely harmful treatment should be stoppable early.

Group sequential boundaries. Pre-specify \(k\) interim analyses and spend the total \(\alpha\) across them using an alpha-spending function. O'Brien-Fleming boundaries are conservative early and near-nominal at the end, which suits the common pattern of wanting an early stop only for a large effect. This requires deciding the number and timing of looks in advance.

Always-valid inference. Use a sequential test whose guarantee holds at every sample size simultaneously, built on mixture sequential probability ratio tests or on confidence sequences derived from time-uniform concentration inequalities. The p-value can be read at any moment, including continuously, and the type I error is controlled over the whole run. This is what most modern experimentation platforms implement, and it is the right default when the interface will be watched.

The cost is explicit and worth stating: always-valid methods are less powerful at any fixed sample size than a correctly executed fixed-horizon test. You pay for the freedom to stop whenever you like, typically in the range of needing somewhat more samples to reach the same power. That is usually a good trade against the alternative of an invalid analysis.

When it breaks

Stopping rules other than significance also bias. Stopping because the effect looks large enough, because a stakeholder asked, or because the quarter ended are all data-dependent if the decision was influenced by the results. Always-valid methods handle stopping at an arbitrary time; they do not handle choosing which metric to report after seeing all of them.

Early estimates are biased upward even when valid. Stopping at the first moment of significance selects on a high realisation of the noise, so the effect estimate at the stopping time is inflated relative to the truth. Sequential tests control the error rate, not the estimate's bias, and reporting the stopping-time point estimate as the expected impact systematically overstates it.

Novelty effects interact badly with early stopping. An effect that is large in the first days and decays is exactly the pattern that triggers an early stop. The sequential machinery is doing its job correctly and the conclusion is still wrong, because the estimand drifted.

Multiple metrics reintroduce multiplicity. Sequential correction handles looking repeatedly at one metric. Looking at twenty metrics, each sequentially valid, has the ordinary multiplicity problem on top, and both corrections are needed.