Online Experimentation
Power, peeking, sample ratio mismatch, variance reduction, interference and long-term effects.
7concepts
142flashcards
51minutes of reading
- 01 Interference and Network Effects in Experiments When one unit's treatment affects another unit's outcome, individual randomisation measures a quantity that is neither the treatment effect nor zero, and the standard designs trade bias against a large loss of power.
- 02 Novelty, Primacy and Long-Term Effects Treatment effects that grow or decay as users learn, how to detect and measure that learning, and the two ways to estimate a long-term effect without waiting years, long-running holdouts and surrogate indices.
- 03 Peeking and Sequential Testing Fixed-sample p-values assume the sample size was chosen in advance, so continuously monitoring a dashboard and stopping at significance can inflate the false-positive rate several-fold.
- 04 Variance Reduction with CUPED and Stratification Regressing out pre-experiment behaviour removes variance that has nothing to do with the treatment, buying sensitivity without extra traffic, and the size of the gain is set by one correlation.