Statistical Power and the Minimum Detectable Effect

A team runs an experiment for two weeks, sees a 1.8% lift, reports \(p = 0.21\), and concludes the feature does nothing. The experiment had 12% power to detect a 2% lift. It was never capable of producing a positive result, and running it was a decision to spend two weeks learning nothing.

Power is \(P(\text{reject } H_0 \mid H_1 \text{ true})\): the probability of detecting an effect that is genuinely there. It is fixed before the experiment by four quantities, any three of which determine the fourth: the significance level \(\alpha\), the effect size, the metric's variance, and the sample size.

The arithmetic, and why it bites

For a two-sided test comparing two means with equal group sizes, the required \(n\) per arm is approximately

At the conventional \(\alpha = 0.05\) and 80% power, \((1.96 + 0.84)^2 \approx 7.85\), so \(n \approx 15.7\,\sigma^2/\delta^2\) per arm.

The term that dominates every practical decision is \(\delta^2\) in the denominator. Halving the effect you want to detect quadruples the sample size. For a relative effect on a rate metric it is worse still: as the baseline rate falls, both the variance term and the absolute effect shrink, and detecting a fixed relative lift on a rarer event needs sample growing faster than linearly in the inverse of the base rate. This is the arithmetic behind the common experience that the first experiments on a surface find large effects easily and every later one is starved.

Concretely: a 5% baseline conversion rate and a target of detecting a 10% relative lift (0.5 percentage points) needs roughly \(15.7 \times 0.05 \times 0.95 / 0.005^2 \approx 30{,}000\) users per arm. Wanting to detect a 5% relative lift instead pushes that to about 120,000.

MDE is the honest way to plan

Rather than asking "is this significant", plan with the minimum detectable effect: given the traffic available and the duration you can afford, what is the smallest effect the experiment can reliably find?

This converts a statistical question into a product one. If the available traffic yields an MDE of 6% and nobody believes the feature will move the metric by more than 2%, the experiment should not be run in that form. The alternatives are real: pick a more sensitive proxy metric, extend duration, reduce variance, or accept a lower confidence level deliberately rather than discovering the underpowering afterwards.

When it breaks

Power calculations assume independent observations. Randomising by user and analysing by event, or randomising by account and analysing by seat, inflates the effective sample by the cluster size and shrinks the computed standard error accordingly. The correct \(n\) is the number of randomisation units, and cluster-randomised designs need a design-effect correction, \(1 + (m-1)\rho\) for clusters of size \(m\) and intra-cluster correlation \(\rho\), which can multiply the requirement several-fold.

Post-hoc power is meaningless. Computing power from the observed effect after a null result is a monotone transformation of the p-value; it adds no information and is routinely misread as evidence that the study was adequate. Power is a pre-experiment design quantity.

Metric variance is not the metric's variance. Revenue per user is heavy-tailed, so \(\sigma^2\) estimated from a short window understates the tail, and the realised power is lower than planned. Winsorising or capping the metric at a pre-specified quantile reduces variance substantially and changes the estimand slightly, a trade worth making explicitly rather than by accident.

Novelty and primacy effects violate the stationarity assumption. The treatment effect in week one is often not the treatment effect in week four, so a duration chosen purely for power can measure a transient. A minimum of one full weekly cycle is standard, and longer where behaviour has a learning curve.

Underpowered experiments do not just fail to detect; they exaggerate. Conditional on reaching significance in a low-power study, the estimated effect must be large, so the published effect is inflated. This is the winner's curse, and it means a portfolio of underpowered experiments produces a set of "wins" whose measured lifts do not replicate.