Mutable Image Tag

Two regions run service:1.8.2. One works and one fails with a missing configuration key. The registry reports a single digest for that tag. Both regions are running what they asked for, and they are running different images.

A tag in most registries is a mutable pointer: pushing the same tag twice leaves two images in storage with the tag on the newer one, and anything that already resolved the tag holds the older digest. Add a pull-through cache with its own expiry, nodes that will not re-pull a layer they already have, and cross-region replication that lands a manifest after the tag moved, and the same reference legitimately means different things in different places at the same moment.

The digest is the identity: a content hash that cannot point anywhere else. A tag is a human label and a cache key. Treating it as an identity produces failures that feel impossible.

Why it matters

It makes the two questions you most need to answer unanswerable. "Is every replica running the same build?" cannot be settled from tags. "What was running in production on the 14^th?" cannot be settled from a deployment log that records a tag, because the tag may have been repointed since.

It also breaks rollback at the worst moment. A rollback to 1.8.1 is a rollback to whatever 1.8.1 currently means, which may be a rebuilt image with a different base layer. A rollback that is not bit-identical to what previously worked is a new deployment with an optimistic name.

Implementation patterns

• Deploy by digest. The pipeline resolves the tag once and writes service@sha256:... into the manifest. This single change removes the entire class, and it makes the deployed artefact provable.
• Enable tag immutability in the registry, so a repush fails loudly instead of silently changing meaning. Most registries support it per repository.
• Carry the tag as a label, not as a reference, so dashboards and humans keep a readable version while machines use the digest.
• Verify replication before rollout. Confirm the digest resolves in every target region before the first region deploys, turning a race into a short wait.
• Record the digest in the deployment event, so the historical question is answered from a log rather than reconstructed.
• Alert on digest uniformity: every replica of a service in an environment running the same digest, outside deployment windows. A few lines against the orchestrator's API, and across 200 nodes it also catches a partial rollout and a node that never pulled.
• Never use latest or a floating major version in anything deployed, and pin base images by digest in the Dockerfile for the same reason one layer down.

Industry example

The industry's answer is visible in the tooling rather than in incident reports. Registries added tag immutability settings; admission controllers exist specifically to reject image references without digests; and Kubernetes' imagePullPolicy semantics are a direct consequence of the ambiguity — IfNotPresent with a mutable tag means a node's cache decides which version runs. Content-addressed storage became the norm in registries from about 2016 precisely so that identity could be separated from naming, and the failures teams still hit are failures to use it.

Failure scenarios

• Two regions or two nodes on different digests for one tag, with behaviour differing and dashboards agreeing.
• A rollback to a rebuilt image, which fails differently from the version it was supposed to restore.
• An audit answer that is wrong rather than missing, because the tag recorded at deploy time now resolves elsewhere.
• A hotfix repushed over a tag while a rollout is in progress, producing a fleet split between two builds with no record.
• A scan result attached to a tag, so the scanned artefact and the running artefact are different — the control passes and protects nothing.
• A pull-through cache serving an old digest long after the registry was updated, which looks like a deployment that silently did not happen.

Trade-offs

Deploying by digest costs readability: a 71-character reference nobody can eyeball, so tooling must carry the tag alongside. It removes the ability to ship a fix by repushing a tag, which is exactly the practice being removed, so teams that relied on it need a fast build-and-deploy path instead. And a digest-pinned manifest must be generated by the pipeline rather than hand-edited, which makes the pipeline the only route to production — a benefit that is experienced as a restriction in the first month.

What it buys is that every question about what ran becomes answerable from records, and that a rollback restores bytes rather than a name.

When not to use it

Where a wrong version is harmless, a tag is fine: a local development compose file, a documentation build, an ephemeral test environment created and destroyed within the hour. The discipline becomes load-bearing as soon as there are multiple regions, a pull-through cache, a gap in time between build and deploy, or an obligation to prove what ran. It is also not a substitute for immutability in the registry — do both, because deploying by digest protects your pipeline's output and immutability protects everyone else's.

Interview question

Q: Two clusters running the same image tag behave differently. Take me through the diagnosis, then tell me how you would make this impossible.

What a strong answer covers: reading the digest each container actually started from rather than trusting the tag; the three mechanisms — a repushed tag, a stale cached tag-to-digest mapping, replication lag — and why the registry's present-tense answer is the misleading signal; deploying by digest resolved once in the pipeline plus registry-side tag immutability; verifying replication before rollout; recording digests in deployment events; the uniformity alert; and the honest cost in readability and in losing the repush habit.

Quick check

Quiz: Why can the registry show one digest for a tag while two regions run different images? Because the registry's answer is present-tense: each region resolved the tag at a different time, and caches or replication can hold an earlier mapping.

Flashcard: What is the identity of a container image? — Its digest. A tag is a mutable pointer and therefore a human label and a cache key, which is why deployments should reference the digest and carry the tag only as metadata.