Container Images for ML and the Build Cache

A Python web service ships in an image of tens of megabytes. A CUDA-enabled PyTorch image ships in double-digit gigabytes: users have reported pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel at 13.2 GB and 2.1.1-cuda12.1-cudnn8-devel at 16.6 GB uncompressed, and traced an earlier jump from 2.76 GB to 5.29 GB to the nvidia-* pip packages pulled in as dependencies (PyTorch forums, Docker image significant size increase). Those are community reports from mixed tags rather than an official table, so treat the exact figures as indicative. The order of magnitude is not in doubt, and everything downstream pays for it: CI minutes, registry egress, autoscaling latency, cold start.

The pull is most of the start

Harter and colleagues measured that pulling package data accounts for 76 percent of container start time while only 6.4 percent of that data is read during startup, and built a lazily-fetching storage driver that cut the median deployment cycle by 5 times (Harter et al., 2016, Slacker: Fast Distribution with Lazy Docker Containers, FAST '16). The idea became production infrastructure as lazy-pull snapshotters such as eStargz and SOCI.

For an ML image the ratio is worse than the paper's workloads, because the image carries compiled kernels for every architecture the wheel supports and your deployment runs on one of them. Most of what you wait for is code for somebody else's GPU.

Layer order is the cache policy

A Dockerfile is a cache invalidation policy written in a declarative syntax. Ordered coarsest-to-finest, the layers are system packages, the CUDA runtime, pinned Python dependencies, model or serving code, entrypoint. A COPY . . placed above the dependency install moves the install behind a layer that changes on every commit, which turns a 4-minute build into a 25-minute one and is the single most common mistake in the file.

Two further mechanics matter. BuildKit cache mounts keep the pip or uv download cache across builds without baking it into a layer, so a changed lock file re-resolves without re-downloading everything. And an unpinned pip install torch against a floating index is not reproducible: the resolution depends on when it ran. A lock file with hashes makes the dependency layer a function of its inputs, which is the precondition for the cache being correct rather than merely fast.

What hides inside a tag

A base image is not inert configuration. NGC's PyTorch containers set TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1, overriding PyTorch's own matmul precision default (MONAI, Precision and Accelerating), so moving a base image tag can change numerical results with no change to the code. The other coupling is the driver boundary: the container carries the CUDA runtime, the host carries the kernel driver, and a wheel built against a newer CUDA than the host driver supports fails at model load, not at build. Both are reasons to pin base images by digest and to assert the driver version at container start.

When it breaks

Ephemeral runners have no cache. A fresh runner per job means a cold layer cache every time, so the cache has to be exported to and imported from a registry, and for ML images that cache is itself gigabytes to transfer. Measure whether the import costs less than the rebuild; for a well-ordered Dockerfile it usually does, and for a badly ordered one it does not help at all.

Slimming the image moves the cost rather than removing it. Keeping the weights out of the image and pulling them from object storage at start is the right default, and it converts a long image pull into a long model download. Under autoscaling, the two are the same number in the p99 of a request that arrived on a cold pod.

--no-cache-dir and cache mounts solve different problems. The flag stops the wheel cache being baked into the layer, which is about image size. The mount stops it being re-downloaded, which is about build time. Teams add the flag, see builds get slower and conclude caching is broken.

Multi-architecture builds double everything. Two platforms means two of every layer, two cache entries and roughly twice the build minutes. Build the second architecture on a schedule rather than per push unless something actually deploys to it.

Image size sets the floor on scale-up. If the pull takes four minutes, no autoscaling policy can respond to a traffic spike in less than four minutes, and the capacity plan has to pre-warm rather than react. That number belongs in the serving SLO document, next to the latency targets it constrains.