Experiment Tracking & Reproducibility
Run metadata, seeds and determinism, environment capture, and what it takes to rebuild a result a year later.
6concepts
67flashcards
43minutes of reading
- 01 Environment Capture and Pinning Why a requirements file does not describe an environment, the layers below Python that also move, and the tradeoff between reproducibility and being able to patch a vulnerability.
- 02 Run Metadata and What to Record The minimum set of facts that makes a training run comparable and rebuildable a year later, why metrics are the least important part, and the discipline that makes tracking survive contact with a deadline.
- 03 Data Versioning and Content Addressing Why versioning datasets is harder than versioning code, how content addressing makes it tractable without copying, and what a dataset version has to mean to be useful.
- 04 Hyperparameter Search Hygiene Why random search beats grid search, how early-stopping schedulers change the budget calculation, and the selection bias that makes the best run's reported score an overestimate.
- 05 Multi-Seed Reporting and Run-to-Run Variance How many seeds a comparison needs, which sources of variance a seed sweep should randomise, and the reporting conventions that stop a one-run improvement from being mistaken for a result.
- 06 Seeds and the Limits of Determinism Why setting a seed does not make a training run reproducible, the specific sources of non-determinism on a GPU, and how to decide how much determinism is worth paying for.