Platforms & Practice
27 min
Four Ways an A/B Test Lies: Power, Peeking, Sample Ratio Mismatch and Interference
At Microsoft, roughly one third of tested ideas move the metric in the intended direction. That number is only meaningful if the tests are trustworthy, and the four failure modes that destroy trustworthiness all produce a clean, …