Seeds and the Limits of Determinism
Why setting a seed does not make a training run reproducible, the specific sources of non-determinism on a GPU, and how to decide how much determinism is worth paying for.
Setting torch.manual_seed(42) produces the comforting impression that the run is now reproducible. Rerun it on the same machine and the loss curve is close but not identical; run it on a different GPU and it diverges visibly by the end. The seed controlled one source of randomness among several, and the others are properties of the hardware and the software stack rather than of the program.
What the seed controls, and what it misses
A seed fixes the pseudo-random number generators used for weight initialisation, dropout masks, data shuffling and augmentation. That is a real and necessary step, and there are usually several generators to set: the framework's, NumPy's, Python's, and one per data-loader worker process, since each worker gets its own.
The sources it does not touch are these.
Non-deterministic kernels. Many GPU reductions accumulate partial results in an order determined by how thread blocks happen to be scheduled, and floating-point addition is not associative, so the sum differs in the last bits between runs. Atomic operations in scatter and index-add kernels are the usual culprits, and they exist because the deterministic alternative is slower.
Autotuning. Libraries that benchmark several algorithms and pick the fastest can pick differently depending on machine load, and different algorithms produce slightly different numerics.
Reduction order in distributed training. All-reduce combines gradients in an order that depends on network timing, so a data-parallel run is non-deterministic across ranks even with identical seeds.
Data-loader ordering. Multiple workers returning batches as they finish means batch order depends on scheduling unless explicitly enforced.
Batch composition. Dynamic batching by token count produces different batches depending on how sequences pack, which is a source of run-to-run difference that seeds cannot address.
Chaos, not error
The differences are in the last bits of a floating-point number, which sounds negligible and is not. Training is a long, non-linear iterated process, so a difference of \(10^{-7}\) at step 1 is amplified through thousands of steps and produces visibly different loss curves by the end. Both runs are correct. Neither is the reference.
This is why "the model got worse after my change" is unanswerable without knowing the run-to-run variance of the unchanged pipeline. Measuring that variance, by running the same configuration several times with different seeds, is the prerequisite for interpreting any experimental result, and it is skipped almost universally.
When it breaks
Full determinism costs throughput. Enabling deterministic algorithms disables the fast non-deterministic kernels and typically costs somewhere between a few percent and a substantial fraction of throughput depending on the model. That is worth paying for debugging and for regression tests, and rarely worth paying for production training runs.
Some operations have no deterministic implementation. Frameworks will raise an error rather than silently producing non-deterministic results when configured to be strict, which is the correct behaviour and means a fully deterministic mode may simply be unavailable for a given model.
Determinism does not survive a hardware change. Different GPU architectures use different kernels and different tensor core paths. Reproducibility is a claim about a specific configuration, and stating it without the hardware is stating half of it.
Chasing bitwise reproducibility is usually the wrong goal. What a team actually needs is that a rerun produces a statistically indistinguishable model. Establishing the variance band and comparing against it is cheaper, more honest, and answers the question people are really asking.
12 flashcards for this concept
Click a card to reveal the answer.