Run Metadata and What to Record
The minimum set of facts that makes a training run comparable and rebuildable a year later, why metrics are the least important part, and the discipline that makes tracking survive contact with a deadline.
A model is in production and performing well. Someone asks what data it was trained on, which hyperparameters produced it, and whether the improvement over the previous version was real or noise. If the answer involves reading a Slack thread from March, the platform has no experiment tracking, whatever tool is installed.
The five things a run must record
Code version. The commit hash, plus an explicit flag for whether the working tree was dirty. A run from uncommitted code is not reproducible and should be recorded as such rather than silently attributed to the last commit.
Data version. An identifier for the exact dataset, ideally a content hash or a table snapshot ID rather than a path, since a path names a location whose contents change. This is the field most often omitted and the one that most often makes a run unrepeatable.
Configuration. Every hyperparameter, including the defaults that were not overridden. Recording only the flags passed on the command line loses the fact that a library's default changed between versions, which is a real and unpleasant class of unexplained difference.
Environment. Library versions, CUDA version, hardware type and count. A run that behaved differently on a different GPU generation is a fact worth being able to establish.
Outputs. Metrics over time, final artefacts, and their storage locations, plus the resource cost. Metrics are the part everyone records and the least useful in isolation, because a metric without its configuration is a number with no meaning.
Automatic beats manual
Tracking that depends on remembering to log is tracking that stops during the week before a deadline, which is exactly when the most consequential runs happen. Everything above can be captured automatically: git state from the repository, configuration from the config object, environment from the interpreter, metrics from callbacks already in the training loop.
The rule that makes this stick is that the tracking is part of the entry point, not part of the training script. If launching a run is impossible without recording it, the discipline requires no discipline.
When it breaks
Recording is not organising. Ten thousand runs with no tags, no grouping and no notion of which experiment they belonged to is an archive nobody searches. A run needs a name for the question it was asking, so results can be grouped and compared against the baseline they were meant to beat.
Artefact storage costs accumulate quietly. Checkpointing every epoch for every run produces terabytes within months. A retention policy, keeping only the best and last checkpoint for runs older than some age and everything for promoted models, is necessary and is almost always added after the first storage bill.
The baseline goes unrecorded. A team compares against "the previous model" without a run ID for it, and when someone asks whether the improvement is real there is nothing to re-evaluate. Every comparison needs both run IDs, and the evaluation data version, or it is not a comparison.
Tracking distributed runs needs a convention. Every rank logging independently produces duplicated or interleaved metrics. Logging from rank zero only is the common choice and hides per-rank divergence, which is exactly the signal you want when a distributed run behaves oddly. Recording per-rank health separately from the run's metrics resolves it.
12 flashcards for this concept
Click a card to reveal the answer.