Your pipeline builds a container image from a tagged commit. Rebuilding the same tag two weeks later produces a different digest. The application behaves identically. Where does the difference come from, and which parts should you fix?

The first three things I would look at

1. Diff the two images layer by layer and then file by file. Not "is it different" but which files differ and how — metadata only, or content.
2. Check the base image reference. If the Dockerfile says a mutable tag, you did not build the same base twice, and everything downstream differs. This is the cause more often than every other item combined.
3. Look at file metadata in the layers you produce. Modification times, ownership and ordering account for most remaining differences when file contents match.

The diagnosis

Differences fall into three classes, and only two are worth work:

Embedded timestamps and paths. Build tooling records when it ran and where: archive member mtimes, .pyc headers, gzip headers, jar manifests, documentation footers, build paths compiled into binaries. Nothing functional changes, and the digest changes. SOURCE_DATE_EPOCH is the conventional fix that a growing number of tools honour, set from the commit's own timestamp, plus a deterministic archiver invocation — sorted entries, fixed ownership, no timestamps.

Non-determinism in ordering. Anything iterating a filesystem directory, a hash map or a parallel worker pool can emit a different order each run: linker input order, an ar archive's member order, a concatenated bundle, a generated file listing. Sort explicitly at every point the order is written down. This is the class that also causes genuinely different behaviour when it interacts with symbol resolution, so it is worth fixing on correctness grounds alone.

Different inputs, which is not a reproducibility problem but a pinning problem. A dependency resolved to a newer version, a base image tag that moved, a generated certificate, a network fetch. Here the rebuild is honestly different because the inputs were different, and the fix is pinning by digest and a lockfile, not determinism flags.

What to fix, and in what order

1. Pin every input by digest — base image, dependencies, toolchain. This is where the safety is, and it is a day of work.
2. Set SOURCE_DATE_EPOCH and use deterministic archiving. Cheap, removes most of the noise, and makes the remaining differences meaningful.
3. Sort anything whose order is written into an artefact.
4. Publish the digest as the identity of a build and promote that digest between environments, so nothing downstream ever needs a rebuild to be identical.

Why it matters even though behaviour is unchanged

Because a non-reproducible build makes "is production running the code we reviewed?" unanswerable. You can prove which digest is deployed and you cannot prove which source produced it, so an audit question becomes an act of faith, and a supply-chain investigation has no baseline to compare against. The xz-utils backdoor disclosed in 2024 lived exactly in the gap between a reviewed tree and a distributed artefact, and a verifying rebuild is the control that closes it.

When this is the wrong investment

Full bit-for-bit reproducibility across a 900-package dependency tree can cost months and may be blocked by a toolchain you do not control. For most teams the 90% that matters is step 1 plus step 4: pin by digest, build once, promote the digest. Choose full reproducibility when you must let a third party verify your artefacts — a regulated supply chain, a security product, an open-source project whose users rebuild — and stop at pinning otherwise.

Common weak answers

• "Use a lockfile." Necessary and not sufficient: a lockfile pins dependency versions and does nothing about timestamps, ordering or a moving base image.
• "Build once and never rebuild." Correct operational advice and it does not answer the question, because the point of a reproducible build is verification by someone who was not there.
• "Compare the source instead." The source was the same in the question. That is precisely what makes the digest difference worth understanding.