Four Ways an A/B Test Lies: Power, Peeking, Sample Ratio Mismatch and Interference

At Microsoft, across a large portfolio of tested ideas, roughly one third produce a positive result, one third produce nothing, and one third actively hurt the metric they were meant to improve (Kohavi and Thomke, 2017, The Surprising Power of Online Experiments, Harvard Business Review, September-October). On heavily optimised surfaces the positive share drops further. The same article describes a Bing headline change that sat in a backlog for months as a low priority, and which on testing raised revenue by 12%, ultimately worth around $100 million.

Both halves of that story depend on the same thing: that the measurements are trustworthy. A portfolio where two thirds of ideas fail is only worth running if the experiments reliably distinguish the third that works. An experiment platform that produces confident, significant, wrong answers is worse than no platform, because it converts an honest "we do not know" into a false "we measured it".

The uncomfortable part is that the four failures that most reliably destroy trustworthiness are not statistical errors in the analysis. They are design and instrumentation failures, they happen before any test is computed, and each of them produces a result that looks exactly like a valid one.

Why this matters: Every one of these failures yields a clean p-value, a tight confidence interval and a plausible effect size. None is detectable by looking harder at the output. Trustworthiness has to be built into the design and the platform, or it is not present at all, and the number of organisations shipping on the basis of experiments that cannot support the conclusion is large.

TL;DR

• Underpowered experiments do not merely fail to detect; they exaggerate. Conditional on reaching significance in a low-power study, the estimated effect must be large, so a portfolio of underpowered tests produces "wins" whose measured lifts do not replicate.
• Sample size scales as \(1/\delta^2\): halving the effect you want to detect quadruples the traffic required. A 5% baseline conversion and a target of detecting a 10% relative lift needs roughly 30,000 users per arm; targeting a 5% lift needs about 120,000.
• Peeking invalidates fixed-sample p-values entirely. Under the null the test statistic is a random walk, and the probability of touching a fixed threshold at some point during a long run rises far above the nominal \(\alpha\) and approaches 1 as monitoring continues.
• A 0.5% deviation from a 50/50 split across 800,000 users has a p-value around \(10^{-6}\). Sample ratio mismatch is a hard stop, not something to reweight, because whatever dropped those users is almost always correlated with the metric.
• Interference means the individually randomised estimate answers the wrong question. It compares "treated in a 50%-treated world" against "untreated in a 50%-treated world", not the launch-relevant "everyone" against "nobody".
• Direction is predictable from the mechanism: competition for a shared resource makes the estimate overstate the global effect; positive spillover makes it understate it.
• The instruments that fix these are all design-time: a pre-registered MDE, always-valid sequential inference, an SRM gate that blocks reporting, and a randomisation unit chosen to contain the interference.
• Variance reduction is the one free lunch: CUPED reduces variance by a factor of \(1-\rho^2\) using pre-experiment data, so a covariate correlated at 0.7 with the metric halves the variance and is worth roughly a doubling of sample size.

At a Glance

flowchart LR
    I["Idea"] --> D["Design:<br/>MDE, unit, duration"]
    D --> A["Assignment"]
    A --> G{"SRM gate"}
    G -->|"split as intended"| M["Measure"]
    G -->|"deviation"| S["STOP: find cause,<br/>fix, rerun"]
    M --> V["Variance reduction<br/>CUPED / stratification"]
    V --> T["Sequential or<br/>fixed-horizon test"]
    T --> R["Decision"]
    D -.->|"interference?"| U["Change the<br/>randomisation unit"]
    U --> A

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

    class I,D blue
    class A,M,T purple
    class G amber
    class S rose
    class V emerald
    class R teal
    class U amber

Three of the four failures are gates in that diagram rather than analysis steps, and the fourth, interference, is resolved before assignment by choosing what gets randomised.

How Online Experimentation Got Its Discipline

Controlled experiments are older than software. Fisher's agricultural work in the 1920s established randomisation, blocking and the analysis of variance, and clinical trials formalised the machinery of blinding, pre-registration and interim monitoring across the following decades. Software inherited almost none of that discipline initially, because the cost structure was different: a web experiment is nearly free to run and can be watched in real time, which removes the constraints that had forced rigour on trials.

The industrial practice was rebuilt over roughly fifteen years, and each step came from a failure that had already happened at scale.

timeline
    title How Online Experimentation Got Its Guardrails
    1935 : Fisher's Design of Experiments establishes randomisation and blocking
    1979 : Pocock and O'Brien-Fleming formalise group-sequential boundaries in clinical trials
    2007 : Kohavi, Henne and Sommerfield publish practical guide to controlled experiments on the web
         : Twyman's law reframed for software: any surprising figure is usually wrong
    2013 : CUPED uses pre-experiment data to cut variance sharply
         : Kohavi et al. describe running controlled experiments at Microsoft scale
    2015 : Johari, Pekelis and Walsh introduce always-valid inference for continuous monitoring
    2019 : Fabijan et al. publish an SRM taxonomy from four companies and over 25 products
    2022 : Kohavi, Deng and Vermeer catalogue the intuitions that survive despite being wrong

The pattern in that timeline is worth naming. Every guardrail exists because a specific, expensive mistake was made repeatedly enough that someone wrote it down. None of them was derived from first principles by people who had not yet been burned.

[IMAGE: Timeline graphic with two parallel tracks. The upper track, "clinical trials", shows Fisher 1935, sequential analysis 1945, group-sequential boundaries 1979, CONSORT 1996. The lower track, "online experiments", shows the 2007-2022 sequence, with dashed arrows from each clinical-trial milestone to its later software rediscovery, annotated with the gap in years. Caption: "Software rediscovered most of trial methodology, roughly forty years late and one incident at a time."]

The Four Failures

One: underpowered experiments exaggerate

Power is the probability of detecting an effect that is genuinely present. For a two-sided comparison of means with equal arms,

At the conventional \(\alpha = 0.05\) and 80% power, \((1.96+0.84)^2 \approx 7.85\), so \(n \approx 15.7\,\sigma^2/\delta^2\) per arm. The whole practical content sits in that squared \(\delta\) in the denominator.

The familiar consequence is that an underpowered test fails to detect real effects. The less familiar and more damaging consequence is what happens when it does reach significance. To clear the threshold with little power, the observed effect must be large, which means it must have been helped by a favourable draw of noise. Conditional on winning, the estimate is biased upward.

The organisational effect compounds. A team running many underpowered experiments accumulates a portfolio of significant wins whose measured lifts are systematically inflated, ships them, and finds the aggregate metric has not moved by anything like the sum of the reported gains. The individual experiments were not fraudulent and the summed forecast was never achievable.

The instrument is the minimum detectable effect, computed before the experiment rather than after:

This converts a statistical question into a product question. Given the traffic available and the time you can afford, what is the smallest effect this experiment could reliably find? If the answer is 6% and nobody believes the change moves the metric by more than 2%, the experiment as designed cannot succeed and the options are: pick a more sensitive proxy metric, reduce variance, extend duration, or decline to run it.

[IMAGE: Two-panel figure. Left: histogram of true effect sizes versus reported effect sizes for experiments that reached significance at 20% power, with the reported distribution shifted right and truncated on the left, shaded to show the inflation region. Right: the same comparison at 80% power, where the two distributions nearly coincide. Caption: "The winner's curse is a function of power, not of honesty."]

Two: peeking breaks the guarantee, not the estimate

A fixed-sample p-value is valid under one condition that is almost never stated: the sample size was fixed before data collection. Choosing when to stop by looking at the results violates it.

Under the null hypothesis the test statistic behaves as a random walk. A fixed threshold is crossed with probability \(\alpha\) at any single pre-specified moment. The probability that a random walk touches that threshold at some point during an extended run is much higher and grows with the number of looks; with unbounded continuous monitoring it approaches 1 even when the true effect is exactly zero.

Johari, Pekelis and Walsh's contribution was to make the problem a platform-design problem rather than a discipline problem (Johari, Pekelis and Walsh, 2015, Always Valid Inference, arXiv:1512.04922; expanded as Johari, Koomen, Pekelis and Walsh, 2017, Peeking at A/B Tests, KDD). If you give experimenters a live significance readout and expect them not to act on it, the process fails; the fix is inference whose guarantee holds at every sample size simultaneously.

Three responses are legitimate and they trade differently:

Fixed horizon, analyse once. Statistically correct, organisationally fragile, and it prevents stopping a harmful treatment early.

Group-sequential boundaries. Pre-specify \(k\) interim analyses and spend the total \(\alpha\) across them. O'Brien-Fleming boundaries spend almost nothing early and approach nominal at the end, which matches the usual intent of stopping early only for something unmistakable. The number and timing of looks must be committed in advance.

Always-valid inference. Confidence sequences derived from time-uniform concentration inequalities, or mixture sequential probability ratio tests, remain valid whenever you read them. This is what modern platforms implement, and it is the right default when the interface will be watched.

The cost of the third option is explicit: less power at any fixed sample size than a correctly executed fixed-horizon test. You are paying for the freedom to stop whenever you like. Against the realistic alternative, which is not a disciplined fixed-horizon test but an undisciplined one, it is a good trade.

One thing no sequential method fixes: stopping at the first moment of significance selects a high realisation of the noise, so the point estimate at the stopping time is inflated. Sequential tests control the error rate, not the estimate's bias, which connects this failure straight back to the first one.

[IMAGE: Line chart with days on the x-axis and running p-value on a log-scale y-axis, showing five simulated null experiments under continuous monitoring. Three of the five dip below the 0.05 line at some point and recover. A separate horizontal band marks an always-valid confidence sequence threshold, which none of the five crosses. Caption: "Under the null, a random walk visits the threshold. A time-uniform bound does not move with it."]

Three: sample ratio mismatch invalidates rather than perturbs

A 50/50 experiment records 402,000 users in control and 398,000 in treatment. That is a 0.5% deviation and it looks like rounding. Under a fair coin across 800,000 users, a chi-squared test on those counts gives a p-value on the order of \(10^{-6}\).

The correct interpretation is not "the split is slightly off". It is that the assignment mechanism did not behave as a fair coin, and whatever caused that is almost certainly correlated with user behaviour.

The instinct to reweight and continue is wrong for a mechanical reason. Consider a treatment that adds a component which loads slowly. Users on poor connections abandon before their assignment is logged. The treatment arm therefore loses a disproportionate share of its least engaged users, and its metric improves for a reason entirely unrelated to the feature. Reweighting corrects the counts and leaves the composition wrong. The two populations are no longer comparable, and no downstream adjustment restores comparability because the missing users are unobserved.

Fabijan and colleagues built a taxonomy of causes from experimentation across four companies and more than 25 products (Fabijan et al., 2019, Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, KDD). Grouping by stage:

sequenceDiagram
    participant U as User
    participant A as Assignment
    participant V as Variant
    participant L as Logging
    participant P as Processing
    U->>A: request
    A->>V: bucket by hash
    Note over A: SRM cause 1 - buggy hash,<br/>variant-dependent eligibility,<br/>carryover from a prior test
    V->>U: render
    Note over V: SRM cause 2 - one variant errors,<br/>times out or redirects<br/>for a subpopulation
    U->>L: emit telemetry
    Note over L: SRM cause 3 - beacon fires at a<br/>different point per variant,<br/>or is blocked more in one arm
    L->>P: batch to warehouse
    Note over P: SRM cause 4 - join, dedup or bot<br/>filter behaves differently because<br/>treatment changed a keyed field
    P->>P: compute metrics

The fourth category is the most insidious, because the experiment executed correctly and the analysis pipeline manufactured the imbalance. Every investigation aimed at the feature comes back clean while the numbers stay wrong.

Two operational notes. The test is extremely sensitive at scale, so at tens of millions of users it will fire on causes that are operationally trivial, and the temptation to loosen the threshold to silence the noise is exactly how a real SRM gets through. And a triggered analysis, restricted to users who reached the feature, will show an imbalance whenever treatment changed who reaches it; the check belongs on the assignment population, not the triggered one, and confusing the two is the commonest false alarm in a mature platform.

[IMAGE: Funnel diagram of an experiment population, 400,000 assigned users per arm narrowing through render, beacon fire, dedup and bot filter stages. The treatment funnel loses an extra 1,260 users at the beacon stage, highlighted, with a callout giving those users' 2.1% historical conversion against the 5.0% population rate. Caption: "The missing users are not a rounding difference; they are a selected subpopulation."]

Four: interference means you measured a different quantity

The first three failures are about whether the comparison is valid. The fourth is about whether the comparison answers the question, and it is the only one that can be entirely correct statistically and still produce a wrong decision.

Writing \(Y_i(1)\) and \(Y_i(0)\) for a unit's potential outcomes presumes that unit \(i\)'s outcome depends only on unit \(i\)'s treatment. That is the "no interference" half of SUTVA, and it fails in any system where units share a finite resource or influence each other: marketplaces, social products, ad auctions, delivery networks, anything with a shared inventory or budget.

Under interference, an individually randomised experiment estimates the difference between "treated in a world that is 50% treated" and "untreated in a world that is 50% treated". The launch decision needs "everyone treated" against "nobody treated". These are different quantities, and the gap between them is not noise.

The direction follows from the mechanism, which makes this the one failure you can reason about before measuring:

• Competition for a shared resource. The treated arm gains partly at the control arm's expense, so control is depressed and the gap widens. The estimate overstates the global effect.
• Positive spillover. Treated users influence untreated ones, lifting control and narrowing the gap. The estimate understates the global effect.

graph TD
    subgraph Ind["Individual randomisation"]
        I1["Treated buyers"] -->|"take supply"| I2["Control buyers"]
        I2 --> I3["Control depressed"]
        I1 --> I4["Measured gap too large"]
        I3 --> I4
    end
    subgraph Clu["Cluster randomisation"]
        C1["Whole region treated"] --> C2["Interference contained<br/>inside the cluster"]
        C2 --> C3["Estimand correct"]
        C2 --> C4["Effective n = number<br/>of clusters"]
    end
    subgraph Swi["Switchback"]
        S1["Whole system alternates<br/>by time period"] --> S2["No simultaneous<br/>cross-arm interference"]
        S2 --> S3["Carryover between periods"]
        S2 --> S4["Confounded with<br/>time of day"]
    end

    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class I1,I2,I3 slate
    class I4 rose
    class C1,C2,C3 emerald
    class C4 amber
    class S1,S2 emerald
    class S3,S4 amber

The designs that contain interference all pay for it in power. Cluster randomisation makes the effective sample the number of clusters: fifty regions is fifty units regardless of user count, and the design effect \(1 + (m-1)\rho\) for clusters of size \(m\) and intra-cluster correlation \(\rho\) can inflate the sample requirement by an order of magnitude. Switchback designs contain interference perfectly within a period and introduce carryover and time confounding. Two-stage saturation designs, which randomise clusters to different treatment fractions and then randomise within them, are the only family that measures the spillover instead of designing around it.

The honest position on a small effect in an interfering system is often that the choice is between a biased estimate and no estimate, and stating that explicitly is better than defaulting to individual randomisation because it fits the power budget.

By the Numbers

Sources: Kohavi and Thomke, 2017, HBR; Fabijan et al., 2019, KDD; Deng, Xu, Kohavi and Walker, 2013, WSDM. The sample-size and SRM p-value rows are computed from the stated formulas rather than quoted from a source, and the Microsoft success-rate figure is a portfolio-level characterisation whose exact value varies by product and maturity; the same authors note that on heavily optimised surfaces such as Bing the positive share is considerably lower.

A Concrete Example

A marketplace tests a new ranking that surfaces newer listings more prominently. Traffic is 240,000 buyers per week. Baseline conversion is 5.0%. Product wants to detect a 10% relative lift.

Step 1: size it. The absolute effect is 0.5 percentage points. With \(\sigma^2 = p(1-p) = 0.05 \times 0.95 = 0.0475\):

Roughly 60,000 users total, so under a week at half the traffic. The team allocates two weeks to cover a full weekly cycle twice, giving about 240,000 per arm and an MDE of about 3.5% relative. Comfortable.

Step 2: apply CUPED. The covariate is each buyer's conversion over the preceding four weeks, correlated with the experiment-period metric at \(\rho = 0.55\) for returning buyers, who are 60% of traffic. Variance falls by \(1 - 0.55^2 = 70\%\) of its original value, that is a 30% reduction, on that segment. New buyers have no pre-period, so they are left unadjusted. Blended, the MDE improves from 3.5% to roughly 3.0% relative. Real, and less dramatic than the headline \(\rho = 0.7\) case.

Step 3: check the split. Day two: 118,240 control, 116,980 treatment. Difference 1,260 on 235,220, or 0.54%. Chi-squared statistic \(\approx (1260/2)^2 / (235220/4) \approx 6.75\) on the standard 50/50 formulation, giving \(p \approx 0.009\); over the full sample by end of week it reaches \(p < 10^{-5}\). The gate fires.

Step 4: find the cause. The new ranking renders an extra image per listing. On slow connections, a share of users abandon before the impression beacon fires. Those users convert at 2.1% historically versus 5.0% overall. Treatment lost about 1,260 of them. Had the experiment been reported: treatment's measured rate excludes 1,260 users who would have converted at 2.1%, adding roughly \(1260 \times (0.05 - 0.021) / 117{,}000 \approx 0.031\) percentage points to treatment's rate, about 0.6% relative, purely from selection. That is a fifth of the effect the experiment was designed to detect, manufactured by a rendering change.

Step 5: fix and rerun. Fire the beacon before the images load. Reruns cleanly, split within tolerance.

Step 6: the result that is still wrong. Two weeks later the experiment reports a 4.2% relative lift in conversion, \(p = 0.003\), no SRM, adequate power, no peeking. The team ships it, and the following month aggregate conversion is up 0.9%, not 4.2%.

Interference. Buyers in the treatment arm were shown newer listings and bought them; those listings then became unavailable to control-arm buyers, who converted slightly less. The measured gap contains both the genuine improvement and a transfer from control to treatment. The follow-up, a region-level cluster randomisation over 40 metropolitan areas, reports a 1.1% lift with a confidence interval of roughly \(\pm\)1.4%, because 40 clusters is 40 units. The cluster design cannot confirm the effect and the individual design cannot be trusted, which is the real position the team is in, and the individually randomised number was never the answer.

[IMAGE: Waterfall chart decomposing the reported 4.2% measured lift into three bars: "true global effect, approximately 1.1%", "cannibalisation from control, approximately 3.1%", and a small "noise" band. A second panel shows the cluster-randomised estimate with its much wider confidence interval overlapping both zero and the true effect. Caption: "The clean estimate was precise about the wrong quantity; the correct design cannot resolve it."]

Where It Breaks

The metric is the assumption nobody audits

Everything above concerns whether the comparison is valid. None of it addresses whether the metric reflects value. Halving the variance of a metric that does not track user benefit makes it easier to detect changes in something that does not matter, and a portfolio of statistically impeccable experiments on a poorly chosen metric optimises the product toward it faithfully.

This is the failure with the worst ratio of consequence to attention, because it produces no anomaly anywhere in the pipeline.

Novelty and primacy make the estimand non-stationary

The treatment effect in week one is frequently not the effect in week four. Users react to change as change, and a new interface produces exploration that decays. An experiment sized purely for power can measure a transient, and worse, the pattern of a large early effect that decays is exactly the pattern that triggers an early stop under sequential monitoring. The machinery works correctly and the conclusion is wrong, because the quantity being estimated moved.

Multiple metrics reintroduce multiplicity on top of everything else

Sequential correction handles repeated looks at one metric. A dashboard with twenty metrics, each sequentially valid, still has an ordinary multiplicity problem, and both corrections are needed. The standard discipline, one pre-registered primary metric plus a small set of guardrails with their own thresholds, exists precisely because correcting twenty metrics into validity leaves the study powered to detect nothing.

Segment analysis is where discipline usually collapses

An experiment that is flat overall gets sliced by platform, country, tenure and device until something is significant. Each slice is a test, the slices were chosen after seeing the aggregate, and the resulting finding has none of its nominal properties. This is the garden of forking paths and it is not repaired by any correction applied afterwards, because the number of paths not taken is unknown.

The organisation is part of the instrument

A platform can implement every guardrail and still produce untrustworthy results if the incentive is to ship. Stopping rules bend, SRM alerts get waived as "probably the bot filter", and the primary metric is chosen after the fact from among those that moved. Kohavi, Deng and Vermeer's catalogue of persistent misunderstandings is partly a document about which wrong intuitions survive contact with organisational pressure (Kohavi, Deng and Vermeer, 2022, A/B Testing Intuition Busters, KDD, 3168-3177).

The structural answer is to make the guardrails properties of the platform rather than of the analyst: an SRM that blocks the results page, a primary metric fixed at experiment creation, sequential inference by default so peeking is harmless.

Alternative Designs

[IMAGE: Decision tree diagram. Root: "can you randomise?" No leads to quasi-experimental designs. Yes leads to "do units interfere?" No leads to standard A/B with a branch on "will anyone watch the dashboard?" selecting sequential versus fixed horizon. Yes leads to a branch on interference mechanism: shared instantaneous supply leads to switchback; social or geographic leads to cluster; wanting to quantify it leads to saturation. Caption: "Four of the eight designs exist only because of interference."]

[IMAGE: Dashboard mockup of an experiment results page in the blocked state. The effect-estimate area is replaced by a banner reading "Sample ratio mismatch detected, p = 3e-7. Results withheld." Below it, the assignment counts, the expected ratio, and a link labelled "diagnostic checklist". Caption: "The guardrail that works is the one that refuses to render a number."]

How It Is Used in Practice

Mature experimentation platforms converge on a similar set of enforced behaviours, and the convergence is informative: these are the things independent organisations arrived at after being wrong.

SRM as a blocking gate. Not a warning, not a badge. The results page refuses to render an effect estimate when the split test fails. Making it advisory reliably produces a culture of waiving it.

A primary metric fixed at creation. The experiment definition records one primary metric and a fixed guardrail set before any data exists. Everything else on the results page is explicitly labelled exploratory and carries no decision authority.

Sequential inference by default. Because the dashboard exists and will be read, the inference must be valid whenever it is read. This is a platform decision made once rather than a discipline demanded of every experimenter.

A/A tests as continuous calibration. Running experiments where both arms receive identical treatment should produce significant differences at the nominal rate and SRMs at the nominal rate. Persistent deviation means the platform is broken, and no experiment running on it can be trusted. This is the only check that validates the instrument rather than a result.

Variance reduction as infrastructure. CUPED applied automatically wherever a pre-period covariate exists, rather than as an analysis someone opts into. The gain is \(1-\rho^2\) and it is free once built.

Interference handled by unit choice, at design time. In marketplaces and social products, the randomisation unit is a standing architectural decision, not a per-experiment one, and the power cost is budgeted for.

The experiment portfolio's value depends on all of this holding simultaneously. One third of ideas working is a good return only if the identification of that third is reliable; if half the wins are artefacts, the portfolio's expected value collapses and the organisation is worse off than if it had shipped on judgment, because it now has false confidence.

Insights Worth Remembering

1. All four failures produce a clean result. A p-value, an interval and a plausible effect. There is no output-side diagnostic that distinguishes a trustworthy experiment from an untrustworthy one, which is why the guardrails have to be design-time and enforced by the platform.

2. Underpowered experiments are not merely uninformative; they are actively misleading. Conditioning on significance in a low-power study selects a favourable noise draw, so reported lifts are inflated and a portfolio of them forecasts an aggregate gain that will never materialise.

3. Peeking is an interface problem. Handing someone a live significance readout and asking them not to act on it is a process design that fails. The fix belongs in the inference, not in the training.

4. SRM is a stop, not an adjustment. The mechanism that unbalanced the counts is almost always correlated with the outcome, so the two populations are not comparable. Reweighting fixes the arithmetic and leaves the composition wrong.

5. Interference changes what you measured, not how precisely you measured it. It is the only one of the four that survives a perfect statistical analysis, and its direction is predictable from whether units compete or reinforce.

6. The power cost of correctness is real and should be stated. Cluster randomisation can inflate the sample requirement by an order of magnitude, and the honest choice is often between a biased estimate and no estimate at all. Choosing individual randomisation because it fits the power budget, without saying so, is the failure.

7. Variance reduction is the only genuinely free improvement here. \(1-\rho^2\) with a pre-experiment covariate costs nothing at experiment time and is worth roughly a doubling of sample at \(\rho = 0.7\). Everything else on this list costs power, traffic or organisational friction.

8. A/A tests validate the instrument; nothing else does. Every other check evaluates a result. Only running the pipeline with a known-null treatment tells you whether the pipeline itself is capable of producing correct answers.

Open Questions

How much of the published effect-size literature survives the winner's curse? It is measured that conditioning on significance inflates estimates, and it is measured that low-power studies are common. What is not established is the magnitude of the aggregate distortion across industry experiment portfolios, because the counterfactual, a full replication of shipped wins, is almost never run.

Can interference be detected rather than assumed? Current practice requires a structural argument about shared resources, or a design that could reveal spillover. Whether a diagnostic exists that flags interference from an individually randomised experiment's own data, without a saturation design, is an open problem and would be a substantial practical advance.

What is the right sequential method when effects are non-stationary? Always-valid inference assumes a fixed underlying effect. Novelty decay violates that, and the interaction between decaying effects and early stopping is understood qualitatively and not well characterised quantitatively.

Do long-term effects justify their measurement cost? Surrogate and holdback methods for estimating long-run impact from short experiments exist, and their validity rests on assumptions about the relationship between short-term and long-term outcomes that are difficult to test without running the long experiment that the method was meant to avoid.

How should experiment portfolios be evaluated as portfolios? Almost all methodology concerns the single experiment. The organisation's actual question is whether the programme produces value, which involves the false discovery rate across the portfolio, the replication rate of shipped wins, and the opportunity cost of experiments that could never have detected their target. That framing is rare and would change which methodological investments look worthwhile.

Sources and Further Reading

1. Kohavi, R., & Thomke, S. (2017). "The Surprising Power of Online Experiments." Harvard Business Review, September-October 2017. HBR
2. Kohavi, R., Deng, A., & Vermeer, L. (2022). "A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments." KDD '22, 3168-3177. DOI
3. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. Publisher
4. Johari, R., Pekelis, L., & Walsh, D. J. (2015). "Always Valid Inference: Bringing Sequential Analysis to A/B Testing." arXiv:1512.04922
5. Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). "Peeking at A/B Tests: Why it matters, and what to do about it." KDD '17, 1517-1525. DOI
6. Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners." KDD '19. DOI
7. Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data." WSDM '13. PDF
8. Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., & Pohlmann, N. (2013). "Online Controlled Experiments at Large Scale." KDD '13. DOI
9. Rubin, D. B. (1974). "Estimating causal effects of treatments in randomized and nonrandomized studies." Journal of Educational Psychology, 66(5), 688-701. DOI
10. Holland, P. W. (1986). "Statistics and Causal Inference." Journal of the American Statistical Association, 81(396), 945-960. DOI
11. Gelman, A., & Loken, E. (2013). "The Garden of Forking Paths." Columbia University working paper. PDF
12. Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012). "Large-Scale Validation and Analysis of Interleaved Search Evaluation." ACM TOIS, 30(1). DOI

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.