Reasoning & Evaluation
24 min
Your Improvement Is Inside the Noise: Seeds, Nondeterminism and the Reproducibility Problem in ML
Change one bit in one weight of a ResNet and, three epochs later, test accuracy differs by more than ten points. Training is a chaotic process, so a seed is not a control variable but a draw from a distribution. Most published an…