Experimentation Ab Testing
5concepts
32flashcards
34minutes of reading
- 01 Sample Ratio Mismatch When the observed traffic split differs from the intended one, the randomisation is compromised and the effect estimate should be discarded rather than adjusted, because the cause is almost always a mechanism that also biases the metric.
- 02 Statistical Power and the Minimum Detectable Effect Sample size is decided by the smallest effect worth detecting, not by convention, and the fourth-power relationship between effect size and required traffic is why most product experiments are underpowered.
- 03 Interference and Network Effects in Experiments When one unit's treatment affects another unit's outcome, individual randomisation measures a quantity that is neither the treatment effect nor zero, and the standard designs trade bias against a large loss of power.
- 04 Peeking and Sequential Testing Fixed-sample p-values assume the sample size was chosen in advance, so continuously monitoring a dashboard and stopping at significance can inflate the false-positive rate several-fold.
- 05 Variance Reduction with CUPED and Stratification Regressing out pre-experiment behaviour removes variance that has nothing to do with the treatment, buying sensitivity without extra traffic, and the size of the gain is set by one correlation.