Multi-Seed Reporting and Run-to-Run Variance

Suppose a new data-augmentation scheme lifts validation accuracy from 81.4 to 81.7 over three seeds each, and the per-seed standard deviation of the pipeline is 0.4 points. The standard error of the difference between two three-run means is \(0.4\sqrt{2/3} \approx 0.33\), so the observed gain is under one standard error. The table in the pull request will still show a bold 81.7. Most of what goes wrong with experimental claims in machine learning happens between that arithmetic and that table.

Seeds and the limits of determinism explains why two runs differ. This concept is about what follows: how many runs a claim needs, what to randomise, and how to report the spread.

How large the noise actually is

Two findings set the scale. In deep reinforcement learning, Henderson and colleagues trained TRPO on HalfCheetah with one fixed hyperparameter configuration and split ten seeds into two groups of five; the two averaged learning curves came from statistically different distributions, with nothing changed but the seed (Henderson et al., 2018, Deep Reinforcement Learning that Matters, AAAI, arXiv:1709.06560). In supervised language tasks, Dodge and colleagues ran 2,100 BERT fine-tuning trials on GLUE that varied only the seed, found that weight initialisation and data order contribute comparably to out-of-sample variance, and saw many runs on small datasets diverge part-way through training (Dodge et al., 2020, Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping, arXiv:2002.06305).

Bouthillier and colleagues then set this noise against the published record. On CIFAR-10 and SST-2, the benchmark variance they measured was of the same order of magnitude as the year-on-year improvements claimed as new state of the art, so many of those increments are not distinguishable from rerunning the old pipeline (Bouthillier et al., 2021, Accounting for Variance in Machine Learning Benchmarks, MLSys, arXiv:2103.03098).

How many seeds

For a two-sided comparison of means with per-run standard deviation \(\sigma\), target difference \(\delta\), significance \(\alpha\) and power \(1-\beta\), the runs needed per arm are approximately

With \(\sigma = 0.4\), \(\delta = 0.3\), \(\alpha = 0.05\) and 80 percent power, \(z\) values of 1.96 and 0.84 give \(n \approx 2 \times 7.84 \times 0.16 / 0.09 \approx 28\) runs per arm. Three is not a small version of that; it is a different experiment, one that can only detect effects around three times the noise. Colas, Sigaud and Oudeyer made the same argument for RL, framing seed count as a power analysis done before the experiment rather than a convention inherited from the last paper (Colas et al., 2018, How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments, arXiv:1806.08295).

Pairing reduces the bill. If run \(i\) of the baseline and run \(i\) of the candidate share a seed for data order and augmentation, the variance of the difference is \(2\sigma^2(1-\rho)\) where \(\rho\) is the within-pair correlation, and a correlated pair can need far fewer runs.

Bouthillier et al. propose a different decision rule: estimate \(P(A > B)\), the probability that one run of \(A\) beats one run of \(B\), as the fraction of paired runs where it did, and declare an improvement only if the confidence interval's lower bound exceeds 0.5 and its upper bound exceeds a threshold \(\gamma\). They recommend \(\gamma = 0.75\), which in their case studies separated benchmark noise from published improvements, and which needs a minimum of 29 paired runs to test reliably.

Randomise more, not less

The same study's counter-intuitive recommendation is to randomise every arbitrary choice between runs: initialisation, data order, augmentation, the train/test split via out-of-bootstrap resampling, even the hyperparameter search. Less correlated runs make the mean over \(k\) runs a better estimate of the pipeline's expected performance. Their cheaper estimator, which holds hyperparameter optimisation fixed, approached the ideal at 51 times less compute. A sweep varying only the initialisation seed understates everything else.

Reporting that survives review

Report the number of runs, the mean or median, and an interval, and say whether the interval is a standard deviation (spread of runs) or a standard error or bootstrap confidence interval (uncertainty in the mean). The two differ by \(\sqrt{n}\) and are routinely confused. When runs occasionally diverge, a mean is dragged by a few failures; the interquartile mean with stratified-bootstrap intervals, proposed for few-run RL benchmarks, is more robust and is what the rliable library implements (Agarwal et al., 2021, Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS, arXiv:2108.13264). Divergence rate is itself a result and belongs in the table.

When it breaks

Seed selection is hyperparameter tuning. Picking the best of five seeds and reporting it is optimisation on the test set with a random-number generator as the search space. The honest report is the distribution, or the expected best-of-\(k\) with \(k\) stated.

The budget does not exist for large runs. Nobody trains a frontier pretraining run 28 times. The substitutes are small-scale proxy sweeps that estimate variance, and treating single large-run comparisons as suggestive. How well small-scale variance transfers is disputed and unresolved.

Test-set noise hides inside seed noise. A fixed test set of 2,000 examples at 80 percent accuracy has a binomial standard error near 0.9 points regardless of seeds. Twenty seeds on one split measure training variance precisely and leave the sampling error of the split untouched, which is why resampled splits belong in the design.

Multiple comparisons return. Ten ablations, each tested at \(\alpha = 0.05\), will produce a false "significant" gain about 40 percent of the time (\(1 - 0.95^{10}\)). A study with many arms needs a correction or a stated exploratory status.