Online Experimentation
Power, peeking, sample ratio mismatch, variance reduction, interference and long-term effects.
7concepts
142flashcards
51minutes of reading
- 01 Overall Evaluation Criterion and Guardrail Metrics Choosing the metric an experiment is judged on, why the obvious business metrics make bad ones, and how guardrail metrics with non-inferiority tests protect everything the goal metric does not measure.
- 02 Sample Ratio Mismatch When the observed traffic split differs from the intended one, the randomisation is compromised and the effect estimate should be discarded rather than adjusted, because the cause is almost always a mechanism that also biases the metric.
- 03 Statistical Power and the Minimum Detectable Effect Sample size is decided by the smallest effect worth detecting, not by convention, and the fourth-power relationship between effect size and required traffic is why most product experiments are underpowered.
- 04 Interference and Network Effects in Experiments When one unit's treatment affects another unit's outcome, individual randomisation measures a quantity that is neither the treatment effect nor zero, and the standard designs trade bias against a large loss of power.
- 05 Novelty, Primacy and Long-Term Effects Treatment effects that grow or decay as users learn, how to detect and measure that learning, and the two ways to estimate a long-term effect without waiting years, long-running holdouts and surrogate indices.
- 06 Peeking and Sequential Testing Fixed-sample p-values assume the sample size was chosen in advance, so continuously monitoring a dashboard and stopping at significance can inflate the false-positive rate several-fold.
- 07 Variance Reduction with CUPED and Stratification Regressing out pre-experiment behaviour removes variance that has nothing to do with the treatment, buying sensitivity without extra traffic, and the size of the gain is set by one correlation.