Environment Capture and Pinning
Why a requirements file does not describe an environment, the layers below Python that also move, and the tradeoff between reproducibility and being able to patch a vulnerability.
An eighteen-month-old training script fails on a fresh machine with an error deep inside a library nobody has heard of. The code is unchanged, the data is unchanged, and something in the environment moved. Reconstructing what it was is archaeology, and the point of environment capture is to make it a lookup instead.
The layers, from loosest to tightest
Unpinned dependencies name packages without versions. This is not an environment description; it is a wish that resolves differently every day.
Pinned direct dependencies name exact versions of what you import. Better, and insufficient, because transitive dependencies still float, and most of the surprising breakages come from a dependency of a dependency.
A full lockfile pins the complete resolved graph with hashes, so the same set of wheels installs every time. This is the minimum bar for a reproducible Python environment, and tools that produce one from a loose specification are what make it maintainable.
A container image, referenced by digest rather than by tag, captures the system libraries, the CUDA runtime, the compiler toolchain and the Python environment together. Tags move; digests do not, and pinning a tag such as cuda:12.4 is a common mistake that reintroduces drift into an otherwise pinned stack.
Below all of this sit the pieces a container does not include: the GPU driver on the host, the kernel, and the hardware itself. Those are recorded rather than pinned, and they belong in the run metadata so a later difference can at least be identified.
The tension nobody resolves cleanly
Perfect pinning means a vulnerability in a transitive dependency stays in every environment until someone deliberately updates. Aggressive updating means environments drift and old results become unreproducible.
The workable arrangement separates the two by lifecycle. Development environments update on a regular cadence with tests catching breakage. Released artefacts pin exactly, and are rebuilt deliberately when a security update requires it, with the rebuild treated as a change that needs validation rather than as a maintenance action. A container image referenced by digest is compatible with both, because rebuilding produces a new digest and the old one remains addressable.
When it breaks
Lockfiles are platform-specific. A lockfile resolved on macOS installs different wheels on Linux, and GPU builds differ from CPU builds of the same package version. Resolution has to happen on the target platform, or the lock describes an environment nobody runs.
Non-Python state escapes capture entirely. Environment variables that change library behaviour, downloaded model weights cached in a home directory, tokeniser files fetched at first use, and configuration read from a mounted path are all inputs and none of them appear in any lockfile. Runs that fetch anything at startup are not reproducible in the strict sense.
Container images expire. A base image referenced by digest can be removed from a registry, and a pinned package version can be yanked from an index. Genuine long-term reproducibility requires storing the artefacts, not just their identifiers, which is a storage cost teams underestimate.
Reproducing the environment does not reproduce the result. Determinism issues remain, so an identical environment gives a statistically identical model rather than a bitwise identical one. Environment capture removes one class of difference; it does not remove all of them, and conflating the two leads to a lot of wasted debugging.
12 flashcards for this concept
Click a card to reveal the answer.