Experiment Tracking & Reproducibility
Run metadata, seeds and determinism, environment capture, and what it takes to rebuild a result a year later.
6concepts
67flashcards
43minutes of reading
- 01 Data Versioning and Content Addressing Why versioning datasets is harder than versioning code, how content addressing makes it tractable without copying, and what a dataset version has to mean to be useful.
- 02 Hyperparameter Search Hygiene Why random search beats grid search, how early-stopping schedulers change the budget calculation, and the selection bias that makes the best run's reported score an overestimate.
- 03 Multi-Seed Reporting and Run-to-Run Variance How many seeds a comparison needs, which sources of variance a seed sweep should randomise, and the reporting conventions that stop a one-run improvement from being mistaken for a result.
- 04 Seeds and the Limits of Determinism Why setting a seed does not make a training run reproducible, the specific sources of non-determinism on a GPU, and how to decide how much determinism is worth paying for.