Your Improvement Is Inside the Noise: Seeds, Nondeterminism and the Reproducibility Problem in ML

Train a ResNet-14 on CIFAR-10 deterministically and it reaches 90.0% test accuracy. Train it again identically, except that one first-layer weight moves to the next representable 32-bit float, a change of about \(5 \times 10^{-10}\). After one epoch the two models differ by 0.18 points. After three epochs they differ by 10.42 points. The perturbed model finishes at 90.12% (Summers and Dinneen, 2021, Nondeterminism and Instability in Neural Network Optimization, ICML, arXiv:2103.04514). Over 100 runs, that one-bit perturbation produced an accuracy standard deviation of 0.21 points, roughly as much as the 0.26 produced by randomising every source of nondeterminism at once.

If a flipped low-order bit is as disruptive as a new seed, the seed was never controlling anything. It selected one trajectory from a chaotic family, all of them the same algorithm. Comparing one trajectory of A with one of B compares two samples of size one, which is what most ML papers and pull requests do.

Why this matters: Every architecture, data and hyperparameter change is justified by a difference between two numbers. When run-to-run noise is as large as the claimed gain, a single-seed comparison is close to a coin toss. Bitwise determinism does not fix this; it only makes the same coin land the same way every time.

TL;DR

• Seeds are samples, not settings. Two groups of five TRPO runs with identical hyperparameters differed at \(p = 0.0016\).
• Every source of nondeterminism is about equally bad, because training is unstable: on CIFAR-10, initialisation, shuffling, augmentation and cuDNN each gave an accuracy SD of 0.22 to 0.25 points; all combined, 0.26.
• Seeds alone move benchmarks. Reseeding BERT fine-tuning beat previously reported BERT results on RTE and CoLA by 7 absolute points.
• Fixed seeds do not remove tooling noise. A WideResNet-28-10 trained 16 times with algorithmic randomness disabled ranged from 77.3% to 80.2%; 83.8% of 901 surveyed practitioners were unaware of or unsure about such variance.
• Determinism is expensive and local. It cost up to 746% overhead on P100s, and PyTorch does not promise it across releases or platforms.
• Five seeds is usually underpowered. Colas et al. put the requirement near 20 seeds per arm for a standardised effect of 1; with 5 seeds, the effect must be 3 to 4 times larger.
• In the worked example, a +1.1 point single-seed "win" becomes a +0.6 mean gain with a 95% interval of \([-0.52, +1.72]\) and \(p = 0.24\).

At a Glance

flowchart LR
  subgraph SRC["Sources of variation"]
    A1["Init and data order"]
    A2["Augmentation and dropout"]
    A3["Kernel and collective order"]
    A4["Split and HPO choices"]
  end
  A1 --> R["One training run"]
  A2 --> R
  A3 --> R
  A4 --> R
  R --> S["One score: a single draw"]
  S --> D["Score distribution over N runs"]
  D --> T["Welch test and interval"]
  T --> C{"Gain outside the noise?"}
  C -->|Yes| SHIP["Claim it"]
  C -->|No| TIE["Report a tie"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class A1,A2,A3,A4 blue
  class R,S purple
  class D,T teal
  class C amber
  class SHIP emerald
  class TIE rose

Before Anyone Ran It Twice

The statistics are older than computers. In 1947 B. L. Welch showed how to compare two means whose populations have different variances, exactly the case of a stable baseline against a jittery new method (Welch, 1947, Biometrika 34(1-2), 28-35). With convex models, retraining found the same optimum, so ML had little reason to care. Deep learning broke that assumption quietly, and the reckoning came in 2017 and 2018.

timeline
    title How ML learned that its seeds were samples
    1947 : Welch generalises the t-test to unequal variances
    2017 : Reimers and Gurevych show seed choice swings NER F1 by one point
         : Melis, Dyer and Blunsom retune LSTMs and beat newer cells
    2018 : Henderson et al. publish Deep RL that Matters
         : Lucic et al. find no GAN consistently beats the 2014 baseline
         : Colas et al. frame seed count as a power analysis
    2019 : NeurIPS runs a reproducibility program and checklist
         : Dodge et al. propose expected validation performance
    2020 : Dodge et al. fine-tune BERT 2,100 times varying only seeds
         : Pham et al. measure implementation-level variance
    2021 : Bouthillier et al. model the whole benchmark process
         : Summers and Dinneen trace nondeterminism to instability
    2022 : Zhuang et al. price deterministic tooling
         : MultiBERTs releases 25 pretraining seeds of BERT-Base

Across 50,000 LSTM sequence taggers, Reimers and Gurevych found the seed alone moved two recent NER systems by one F1 point, enough to be perceived "either as state-of-the-art or mediocre" (Reimers and Gurevych, 2017, EMNLP, arXiv:1707.09861). Melis, Dyer and Blunsom retuned plain LSTMs to 58.3 test perplexity on Penn Treebank at 24M parameters, against 62.4 published for the 54M NAS cell; seeds plus floating-point ordering alone were worth about 0.4 (Melis et al., 2018, ICLR, arXiv:1707.05589). A neutral GAN study "did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN" of 2014 (Lucic et al., 2018, NeurIPS, arXiv:1711.10337).

The most quoted demonstration came from RL. Henderson et al. trained TRPO on HalfCheetah ten times with one configuration, split the runs into two sets of five and averaged each; a t-test across training gave \(t = -9.09\), \(p = 0.0016\) (Henderson et al., 2018, Deep Reinforcement Learning that Matters, AAAI, arXiv:1709.06560). Nothing differed but seeds.

[IMAGE: Two TRPO learning curves on HalfCheetah, each the mean of five seeds with a shaded band, visibly separating as training proceeds. Caption: "Same algorithm, same hyperparameters, different seeds: t = -9.09, p = 0.0016 (Henderson et al., 2018)."]

NeurIPS 2019 added a reproducibility checklist asking for clearly defined error bars. Code at camera-ready rose from under 50% to 74.4%, yet 36% of submissions judged error bars not applicable to their results (Pineau et al., 2021, JMLR, arXiv:2003.12206). Code became common. Variance reporting did not.

How Run-to-Run Variance Actually Works

Three properties called "reproducibility"

Bitwise determinism (identical weights from identical code, data, seed, stack and hardware) is a debugging property. Distributional replicability (fresh randomness reproduces your score distribution) and claim robustness ("A beats B" survives re-running) are what support a claim; a deterministic pipeline that drew a lucky trajectory reproduces its lucky number forever. Determinism makes a result repeatable; only sampling makes it trustworthy.

Where the randomness enters

Write a run as \(\theta_T = \mathcal{A}(D, h, \xi)\), with \(\xi\) bundling every source of randomness. Bouthillier et al. varied each part of \(\xi\) (data sampling, augmentation, initialisation, visit order, dropout, numerical noise, hyperparameter search) across five case studies (Bouthillier et al., 2021, Accounting for Variance in Machine Learning Benchmarks, MLSys, arXiv:2103.03098). Bootstrapping the train/test split "stands out as the most important source of variance." Initialisation was generally under 50% of it, on par with visit order, and HPO induced about as much as initialisation. The seed people usually vary is a minority of the variance, and sources do not add: total variance "cannot be obtained by simply adding them up."

Summers and Dinneen varied one source at a time over 100 runs of ResNet-14 on CIFAR-10:

Every row is essentially the same number, and models differing only in cuDNN ordering disagree on more than one test image in ten.

Why every source is equally bad: instability

Training amplifies small differences. A useful caricature (a modelling assumption, not a result from these papers) has the gap between two trajectories grow exponentially early in training:

where \(\lambda > 0\) acts as a Lyapunov exponent for the optimiser. From \(5 \times 10^{-10}\), the gap reaches weight scale after about \(\ln(10^{9})/\lambda \approx 21/\lambda\) steps, and the origin of the perturbation is forgotten. Every source feeds the same amplifier and emerges as the same spread. Summers and Dinneen's control fits: for a linear model the one-bit change gave an accuracy SD of exactly 0, while one hidden layer raised it to 0.28 (fully connected) or 0.14 (convolutional).

So eliminating one source of nondeterminism does not reduce variance; whatever remains is amplified to the same level. Zhuang et al. agree from the tooling side: "Non-determinism must be controlled at all levels of the technical stack or is not worth controlling at all" (Zhuang et al., 2022, Randomness in Neural Network Training: Characterizing the Impact of Tooling, MLSys, arXiv:2106.11872).

[IMAGE: Log-scale plot of weight distance between two runs versus training step, starting at 5e-10 after a one-bit change, rising linearly, then flattening at the independent-seed level (dashed line). Caption: "Instability erases the origin of a perturbation."]

The GPU layer

Implementation noise survives "fixing all the seeds." With cudnn.benchmark on, cuDNN times several convolution algorithms, and PyTorch warns that "the benchmark may select different algorithms on subsequent runs, even on the same machine" (PyTorch 2.14, Reproducibility). Different algorithms sum in different orders, and floating-point addition is not associative. Operations such as scatter_add_ and index_add write many values into one slot in parallel; torch.use_deterministic_algorithms(True) swaps in deterministic versions or raises a RuntimeError, for example for histc on CUDA (PyTorch, torch.use_deterministic_algorithms). NCCL chooses its reduction algorithm itself; forcing it with NCCL_ALGO sits among debugging variables that "should not be used in production" (NVIDIA, NCCL environment variables). Change the node count and every gradient's summation order changes.

None of these is a bug; they are the price of fast kernels. Pham et al. measured it: with every algorithmic source fixed, 16 runs of WideResNet-28-10 on CIFAR-100 on one RTX 2080 Ti ranged from 77.3% to 80.2% (Pham et al., 2020, ASE).

The statistics a comparison needs

Run A \(n_A\) times and B \(n_B\) times, with means \(\bar{y}_A, \bar{y}_B\) and variances \(s_A^2, s_B^2\). The target is \(\Delta = \mu_B - \mu_A\), the difference in expected score of two pipelines. Welch divides the observed difference by its unpooled standard error:

Not pooling matters because new methods are often less stable than tuned baselines. The cost is fractional degrees of freedom, from the Welch-Satterthwaite approximation:

When one arm's variance dominates, \(\nu\) collapses toward that arm's \(n - 1\): a five-seed comparison against a noisy method has nearer 4 degrees of freedom than 8, and the 95% critical value rises from 2.31 toward 2.78. The interval for \(\Delta\) is \((\bar{y}_B - \bar{y}_A) \pm t_{0.975,\nu} \cdot \mathrm{SE}\).

Which test to trust at small \(n\) is empirical. Colas, Sigaud and Oudeyer simulated six tests on normal, bimodal and log-normal distributions and on 192 real runs each of SAC and TD3 (Colas et al., 2019, A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms, arXiv:1904.06979). The bootstrap test "should never be used for sample sizes below N = 50," the permutation test not below 10, and rank tests misfire when one distribution is skewed. Student's and Welch's t-tests were the most robust, Welch slightly more so, though its false-positive rate still reached about 0.1 under skew; hence their advice to test at \(\alpha = 0.01\) when in doubt.

How many seeds

Power sets the seed budget. With standardised effect \(\epsilon = |\Delta| / \sigma_{\text{pool}}\) and \(\sigma_{\text{pool}} = \sqrt{(\sigma_A^2 + \sigma_B^2)/2}\), the normal approximation for runs per arm is

about 16 at \(\epsilon = 1\) and 63 at \(\epsilon = 0.5\); the exact t-based calculation adds one or two. Colas et al.'s coarse-grid simulations recommend about 20 seeds for \(\epsilon = 1\), about 100 for \(\epsilon = 0.5\) ("more than 50" in their results section), and 5 to 10 for \(\epsilon = 2\). Real gains, a few tenths of a point against an SD of a few tenths, sit at \(\epsilon\) between 0.5 and 1. Five seeds is a budget for effects nobody needs a test to see.

Pairing helps when legitimate. If run \(i\) of each arm shares its data-order seed, \(\mathrm{Var}(Y_B - Y_A) = \sigma_A^2 + \sigma_B^2 - 2\rho\,\sigma_A \sigma_B\), and positive \(\rho\) shrinks it; instability erodes \(\rho\), so measure it before relying on it.

Beyond the mean

Bouthillier et al. ask instead how often A beats B, estimated from \(k\) paired runs:

An improvement is "significant and meaningful" when the lower confidence bound exceeds 0.5 and the upper bound exceeds \(\gamma\); their recommended \(\gamma = 0.75\) needs 29 runs to detect reliably. Dodge et al. target unequal tuning: expected validation performance is the expected best of \(n\) trials, from the empirical CDF \(\hat{F}\) of sorted scores \(v_{(j)}\):

so methods are compared as curves over budget (Dodge et al., 2019, Show Your Work, EMNLP, arXiv:1909.03004). It also prices seed shopping.

[IMAGE: Overlapping histograms of 25 seeds per arm, A centred at 81.2 and a wider B at 81.8, with each arm's best seed marked. Caption: "Mean gap, best-seed gap and P(B beats A) are three different claims."]

Seeing It in Motion

Why fixing seeds leaves a residue: much of the randomness sits where seeds do not reach.

flowchart TB
  subgraph ALG["Algorithmic, controlled by seeds"]
    I["Weight init"]
    O["Minibatch order"]
    AU["Augmentation"]
    DR["Dropout masks"]
  end
  subgraph IMP["Implementation, not seeded"]
    CU["cuDNN algorithm choice"]
    SC["Scatter and index-add writes"]
    CO["NCCL reduction order"]
    CP["Compiler autotuning"]
  end
  subgraph EXP["Experimental design"]
    SP["Train and test split"]
    HP["Hyperparameter search"]
  end
  ALG --> AMP["Instability amplifies everything"]
  IMP --> AMP
  AMP --> VAR["Run-to-run variance"]
  EXP --> VAR
  VAR --> CL["Your claimed gain"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  class I,O,AU,DR,SP,HP blue
  class CU,SC,CO,CP slate
  class AMP purple
  class VAR amber
  class CL rose

A claim should move through a planned lifecycle rather than being discovered in one run:

stateDiagram-v2
    [*] --> Observed: one run beats baseline
    Observed --> Planned: set effect size and power
    Planned --> Replicated: N seeds per arm
    Replicated --> Tested: Welch t and interval
    Tested --> Supported: interval excludes zero
    Tested --> Inconclusive: interval spans zero
    Inconclusive --> Planned: add seeds if gain matters
    Inconclusive --> Tie: report no difference
    Supported --> Robust: holds on resampled splits
    Robust --> [*]
    Tie --> [*]

The Inconclusive loop is where discipline fails: adding seeds until \(p < 0.05\) is optional stopping, which inflates false positives. Fix the seed count before the first run.

By the Numbers

Sources: Henderson et al.; Dodge et al., 2020; Summers and Dinneen; Pham et al.; Picard; Zhuang et al.; Agarwal et al.. Figures hold for each paper's hardware and software; they are not transferable constants.

Picard's 10,000-seed scan clustered between 89.5% and 90.5%, yet its extremes were 1.82 points apart, a gap "widely considered as significant" (Picard, 2021, torch.manual_seed(3407) is all you need, arXiv:2109.08203). A modest seed search finds a 1-point win by accident. And on V100, Zhuang et al.'s implementation-only noise exceeded algorithm-only noise.

[IMAGE: Horizontal bars on one "accuracy points" axis: run-to-run SD (0.26, amber), min-max over 10,000 seeds (1.82, amber), fixed-seed implementation range (2.9, amber), beside a band for typical claimed gains (emerald). Caption: "The yardstick and the thing being measured are the same length."]

A Concrete Example

The numbers are illustrative but typical of a CIFAR-scale classifier. A team proposes warmup schedule B against baseline A. The pull request reports seed 0: A = 81.2%, B = 82.3%, +1.1 points. A reviewer asks for seeds 0 to 4 on both arms.

Step 1: scores.

Step 2: means and variances. \(\bar{y}_A = 406.0/5 = 81.2\); deviations \(0, -0.6, +0.7, -0.3, +0.2\) square-sum to \(0.98\), so \(s_A^2 = 0.245\). \(\bar{y}_B = 409.0/5 = 81.8\); deviations \(+0.5, -0.8, +1.0, -1.1, +0.4\) square-sum to \(3.26\), so \(s_B^2 = 0.815\). The mean gain is +0.6, not +1.1, and B's variance is 3.3 times A's.

Step 3: standard error and t. \(0.245/5 = 0.049\) and \(0.815/5 = 0.163\) sum to \(0.212\), so \(\mathrm{SE} = 0.460\) and Welch's \(t = 0.6 / 0.460 = 1.30\).

Step 4: degrees of freedom. \(\nu = 0.212^2 / (0.049^2/4 + 0.163^2/4) = 0.0449 / 0.00724 \approx 6.2\), below the 8 a pooled test assumes, because B's noise dominates.

Step 5: p-value and interval. At \(\nu = 6.2\), two-sided \(p \approx 0.24\). With \(t_{0.975, 6.2} \approx 2.43\), the interval is \(0.6 \pm 2.43 \times 0.460 = [-0.52, +1.72]\): B could be half a point worse or 1.7 better.

Step 6: paired view. Differences \(+1.1, +0.4, +0.9, -0.2, +0.8\) have mean \(0.6\) and SD \(0.515\), so paired \(t = 0.6/(0.515/\sqrt{5}) = 2.61\) on 4 df, \(p \approx 0.06\). Shared seeds induced correlation (0.89 here), which helped, but not past 0.05.

Step 7: probability of outperforming. B wins 18 of the 25 cross-pairs, \(\hat{P}(B > A) = 0.72\), and 4 of 5 same-seed pairs. Bouthillier et al.'s bar needs about 29 runs.

Step 8: could the experiment answer at all? Pooled SD is \(\sqrt{(0.245 + 0.815)/2} = 0.728\), so \(\epsilon = 0.82\). Five seeds per arm gives about 21% power; 80% needs about 25 per arm (exact t; the normal approximation gives 23).

Verdict. "B: 81.8 ± 0.9 (SD, n = 5) vs A: 81.2 ± 0.5; difference +0.6, 95% CI \([-0.5, +1.7]\), Welch \(p = 0.24\); underpowered, inconclusive." If 0.6 points matters, plan 20 more seeds per arm now, not one at a time until the p-value dips.

[IMAGE: Dot plot of the five A and five B scores from 80.5 to 83.0, thin lines joining same-seed pairs, bars at each mean, and the Welch 95% interval for the difference crossing a dashed zero line. Caption: "The single-seed claim was +1.1. The interval runs from -0.5 to +1.7."]

Where It Breaks

Seed shopping is search on the test set

A crashed run is relaunched with a new seed; the better rerun becomes the number. That is a search over seeds with the test set as objective. Dodge et al. show the magnitude: seeds alone lifted RTE and CoLA 7 points above previously reported BERT numbers, changing "never the fine-tuning regime, hyperparameter values, or pretrained weights" (Dodge et al., 2020, Fine-Tuning Pretrained Language Models, arXiv:2002.06305). On small datasets many trials diverged partway, so a five-seed mean depends on whether a divergent run landed in the sample. Declare seeds in advance and report the divergence rate.

Determinism does not travel, and it is not free

PyTorch states that "completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms." A deterministic run on eight GPUs is not bitwise reproducible on sixteen, so a stored "golden" loss curve breaks at the first infrastructure upgrade, while a distributional test survives it. Zhuang et al. also measured deterministic-mode overhead for 7x7 convolutions of up to 746% on P100, 241% on V100 and 196% on T4. That budget buys two to seven extra ordinary runs, which say far more about whether a gain is real.

Test-set noise hides inside seed noise

Bouthillier et al. found that data-sampling variance matched a binomial model, \(\mathrm{SD} \approx \sqrt{p(1-p)/n'}\). For GLUE RTE (277 validation examples, about 66% accuracy) that is about 2.8 points; for CIFAR-10 (10,000 images, 91%) about 0.29. Twenty seeds on one split leave that untouched (see Error Bars for Evals).

Mean accuracy hides where the variance went

Zhuang et al. found top-line accuracy barely moved while underrepresented CelebA subgroups were "disproportionately impacted by the introduction of stochasticity." A 10.7% disagreement rate means two equally accurate models disagree on about a thousand CIFAR-10 images. D'Amour et al. call this underspecification: predictors equivalent on held-out data behave differently under shift (D'Amour et al., 2020, arXiv:2011.03395).

[IMAGE: Grid of 100 random CIFAR-10 test images, each tile split into two halves coloured by the prediction of two ResNet-14 runs with identical accuracy; about eleven tiles show mismatched colours. Caption: "Same accuracy, different models: roughly one prediction in ten flips between retrains."]

Tuning budgets and skewed distributions

Melis et al. and Lucic et al. found advantages shrank once baselines got comparable search. RL returns and divergence-prone fine-tuning are skewed or bimodal; Agarwal et al. recommend the interquartile mean with stratified bootstrap intervals (Agarwal et al., 2021, Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS, arXiv:2108.13264), which collides with Colas et al.'s warning about bootstrap tests below \(n \approx 50\), exactly where most teams operate. And a ten-row ablation table tested at \(\alpha = 0.05\) has a \(1 - 0.95^{10} = 40\%\) chance of at least one spurious win.

Alternative Designs

These are layers more than rivals. The live disagreement is the decision rule. Colas et al. recommend Welch with a conservative \(\alpha\). Agarwal et al. recommend against significance tests because of their "dichotomous nature" and common misreadings, preferring intervals. Bouthillier et al. propose \(P(A > B)\) with a threshold. All agree single-run point estimates are not evidence. For ship decisions, a pre-registered rule works: the lower 95% bound must exceed zero, or the minimum gain worth the complexity.

How It Is Used in Practice

Record enough to re-run and to analyse. Make each seed a child of a parent run holding the plan, so nobody quietly drops a bad seed. MLflow tags runs automatically with mlflow.source.git.commit, mlflow.docker.image.name and mlflow.parentRunId, and logs datasets with mlflow.log_input (MLflow, Tracking API). Add every seed used (including data-loader workers), CUDA, cuDNN and NCCL versions, GPU model and world size, determinism flags, a data and split hash, the HPO budget, and every launched run.

Deterministic CI, stochastic experiments. A short deterministic job in CI catches changes that alter numerics; PyTorch notes determinism "may save time in development by facilitating experimentation, debugging, and regression testing."

Budget seeds before running. Estimate the SD from an existing sweep, pick the smallest gain worth shipping, compute \(n\), and label the result exploratory if \(n\) is unaffordable. Agarwal et al.'s estimate of over 1,000 GPU days for five runs on 50-plus Atari games explains why RL moved to interval-based aggregates.

Pay for stability when churn matters. A five-model ensemble cut accuracy SD from 0.26 to 0.12 and disagreement from 10.7% to 4.1% at five times the cost (Summers and Dinneen).

Insights Worth Remembering

1. All nondeterminism is the same nondeterminism. A one-bit change, a reshuffle and a cuDNN algorithm switch give the same variance because training amplifies any perturbation. Removing one source while others remain buys nothing.

2. The largest variance is usually not the seed you vary. Data sampling dominated in Bouthillier et al.; init-seed sweeps understate a claim's uncertainty.

3. Determinism is for debugging; sampling is for evidence. Deterministic mode costs compute, holds on one stack, and says nothing about whether a gain is real.

4. Five seeds detect only effects nobody needed a test for. At \(\epsilon = 0.8\), five seeds per arm gives about 21% power. "Inconclusive" is the honest outcome of many five-seed comparisons.

5. Best-of-k is a different statistic from the mean. The expected maximum rises with the number of tries. If you searched, report the search.

6. Equal accuracy does not mean equal models. Same-score runs disagree on about one CIFAR-10 image in ten, and subgroups absorb disproportionate variance. Track churn.

Open Questions

Does small-scale variance transfer to large scale? Nobody trains a frontier model 25 times, so teams rely on proxy runs; the 25 MultiBERTs checkpoints give seed-level evidence at BERT-Base scale (Sellam et al., 2022, ICLR, arXiv:2106.16163), but no systematic study we could verify shows small-model variance predicting large-model variance.

Can training be made less chaotic without ensembling? Instability drives variance, and Zhuang et al. measured batch normalisation cutting a small CNN's accuracy SD from 0.86 to 0.30. Which other design choices reduce instability, and why, is not characterised.

Which decision rule should become the default? Welch tests, intervals and \(P(A > B)\) can disagree on the same data, and the field has not converged.

How much benchmark progress is variance? Bouthillier et al. found CIFAR-10 and SST-2 benchmark variance on the order of published increments. We found no multi-seed re-audit of a modern leaderboard, so the fraction of progress that is noise remains an estimate.

Sources and Further Reading

1. Welch, B. L. (1947). "The Generalization of 'Student's' Problem when Several Different Population Variances are Involved." Biometrika, 34(1-2), 28-35. doi:10.1093/biomet/34.1-2.28
2. Henderson, P., Islam, R., Bachman, P., et al. (2018). "Deep Reinforcement Learning that Matters." AAAI 2018. arXiv:1709.06560
3. Bouthillier, X., Delaunay, P., Bronzi, M., et al. (2021). "Accounting for Variance in Machine Learning Benchmarks." MLSys 2021. arXiv:2103.03098
4. Summers, C., & Dinneen, M. J. (2021). "Nondeterminism and Instability in Neural Network Optimization." ICML 2021. arXiv:2103.04514
5. Colas, C., Sigaud, O., & Oudeyer, P.-Y. (2019). "A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms." arXiv:1904.06979
6. Colas, C., Sigaud, O., & Oudeyer, P.-Y. (2018). "How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments." arXiv:1806.08295
7. Dodge, J., Ilharco, G., Schwartz, R., et al. (2020). "Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping." arXiv:2002.06305
8. Dodge, J., Gururangan, S., Card, D., et al. (2019). "Show Your Work: Improved Reporting of Experimental Results." EMNLP 2019. arXiv:1909.03004
9. Melis, G., Dyer, C., & Blunsom, P. (2018). "On the State of the Art of Evaluation in Neural Language Models." ICLR 2018. arXiv:1707.05589
10. Lucic, M., Kurach, K., Michalski, M., et al. (2018). "Are GANs Created Equal? A Large-Scale Study." NeurIPS 2018. arXiv:1711.10337
11. Reimers, N., & Gurevych, I. (2017). "Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging." EMNLP 2017. arXiv:1707.09861
12. Pineau, J., Vincent-Lamarre, P., Sinha, K., et al. (2021). "Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)." JMLR. arXiv:2003.12206
13. Pham, H. V., Qian, S., Wang, J., et al. (2020). "Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance." ASE 2020. PDF
14. Zhuang, D., Zhang, X., Song, S. L., & Hooker, S. (2022). "Randomness in Neural Network Training: Characterizing the Impact of Tooling." MLSys 2022. arXiv:2106.11872
15. Agarwal, R., Schwarzer, M., Castro, P. S., et al. (2021). "Deep Reinforcement Learning at the Edge of the Statistical Precipice." NeurIPS 2021. arXiv:2108.13264
16. Picard, D. (2021). "torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision." arXiv:2109.08203
17. Sellam, T., Yadlowsky, S., Wei, J., et al. (2022). "The MultiBERTs: BERT Reproductions for Robustness Analysis." ICLR 2022. arXiv:2106.16163; D'Amour, A., Heller, K., Moldovan, D., et al. (2020). "Underspecification Presents Challenges for Credibility in Modern Machine Learning." arXiv:2011.03395
18. Documentation: PyTorch 2.14, Reproducibility and torch.use_deterministic_algorithms; NVIDIA, NCCL environment variables; MLflow, Tracking API.

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.