Overall Evaluation Criterion and Guardrail Metrics

A bug in one of Bing's experiments served users very poor search results. Two metrics the company tracked progress by improved significantly: distinct queries per user rose by more than 10%, and revenue per user by more than 30% (Kohavi et al., 2012, Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained, KDD). Worse results forced people to search again and to click ads instead of organic links. Judged on those metrics, the broken treatment was a triumph. The experiment was fine; the scoreboard was wrong.

What an OEC has to do

The overall evaluation criterion is the quantitative measure an experiment's success is decided on, agreed before the experiment starts (Kohavi, Longbotham, Sommerfield and Henne, 2009, Controlled Experiments on the Web: Survey and Practical Guide, Data Mining and Knowledge Discovery 18(1)). It must be measurable within an experiment's duration, which is usually a few weeks, and it must be causally aligned with the long-term goal the business actually has. Those two demands pull in opposite directions, and most OEC design is managing that tension.

The Bing case shows how a long-term goal decomposes into short-term levers pointing in opposite directions. Query share's numerator can be written as

A better search engine should raise sessions per user, since satisfied users come back, and lower queries per session, since users find answers faster. Queries per user mixes the two, so it rewards frustration. Kohavi and colleagues concluded that distinct queries alone should not be used as an OEC for search.

Deng and Shi name the two properties an OEC needs (Deng and Shi, 2016, Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned, KDD). Directionality: movement has an unambiguous interpretation in terms of user value. Sensitivity: it moves detectably when user value changes. They decompose sensitivity as

the probability a change truly moves the metric times the power to see it. The decomposition diagnoses different failures. Sessions per user is insensitive mainly because real changes rarely move it within weeks, a low \(P(H_1)\). Revenue per user is insensitive because its skewed distribution kills power, which variance reduction can fix. Directionality is validated with a corpus of past experiments whose verdicts are trusted, and with deliberate degradation experiments, since it is far easier to make a product measurably worse than better.

Guardrails

No single metric captures everything a change can break. Guardrail metrics watch the dimensions the OEC ignores: page latency, crash rate, unsubscribes, revenue when the OEC is a satisfaction metric, satisfaction when it is revenue. Deng and Shi separate them from debugging metrics, which explain why the OEC moved and need sensitivity more than clear direction. Trust guardrails such as sample ratio mismatch, covered in their own concept, are a third category: they say whether the experiment can be believed at all.

A guardrail asks a different statistical question from the OEC. The OEC asks "did it improve"; a guardrail asks "did it get unacceptably worse". The right test is non-inferiority with a margin \(\delta > 0\):

Testing a guardrail for a significant difference is the common mistake: an underpowered guardrail that shows no significant regression is read as safe when it was never able to see one.

Spotify's decision-rule framework makes the multiple-metric logic explicit (Schultzberg, Ankargren and Frånberg, 2024, Risk-Aware Product Decisions in A/B Tests with Multiple Metrics, arXiv:2402.11609). If shipping requires the success metric to improve and every non-inferiority guardrail to pass, the false-positive rate does not need a multiplicity correction for the guardrails, because every test must reject. Power does need correcting. With one success metric and four independent guardrails each powered at 80%, the chance that all five come out right when the change is genuinely good is \(0.8^5 \approx 0.33\). Reaching 80% for the decision needs each test at \(0.8^{1/5} \approx 0.956\) power, which is a much larger experiment.

When it breaks

Single OEC or decision rule is a live disagreement. Kohavi and colleagues argue for agreeing one OEC in advance, if necessary a weighted combination of metrics, so that trade-offs are settled before results arrive. The decision-rule school argues that weights between, say, latency and engagement are rarely known and that explicit success and non-inferiority conditions are more honest. Both reject the common practice of choosing which metric mattered after seeing the results.

Every OEC becomes a target. Teams optimise what they are scored on. An OEC built on clicks invites clickbait; one built on time spent invites friction. Periodic re-validation against degradation experiments and long-term holdouts is the defence, and it has to be budgeted as ongoing work.

Rate metrics can move for the wrong reason. A click-through rate rises if the treatment reduces the denominator. Always read a ratio OEC alongside its numerator and denominator counts.

Guardrail margins are policy, not statistics. Whether a 1% latency regression is acceptable depends on what it costs, and a margin chosen to make an experiment pass is not a guardrail. Margins should be set per metric, in advance, by whoever owns the downside.