A deployment in one region runs correctly and the same deployment in another fails with a missing configuration key. Both reference image `service:1.8.2`. The registry reports one digest for that tag. How do you diagnose this, and what is the fix?

The first three things I would look at

1. The digest each running container was actually started from, not the tag it asks for. Every orchestrator records it, and if the two digests differ, the investigation is over in a minute and the remaining question is why.
2. Whether the tag was ever repushed. A tag is a mutable pointer in most registries: pushing 1.8.2 twice leaves two images, the tag on the newer one, and anything that already resolved the tag holding the older digest.
3. The pull path in the failing region — a mirror, a pull-through cache or a regional replica. Each is a place where a tag-to-digest mapping is cached with its own expiry.

The diagnosis

The mechanism is nearly always one of three, and they compound:

• A repushed tag. Someone fixed the image and pushed the same tag. Region A pulled before, region B after, and both believe they run 1.8.2. This is the most common cause and it is a process failure, not a registry bug.
• A stale cached mapping. A pull-through cache or mirror resolved the tag earlier and serves the old digest until its entry expires. Nodes that already have a layer cached with imagePullPolicy: IfNotPresent never re-resolve at all.
• Replication lag. A manifest replicated to the second region after the tag was updated, so for a window the tag resolves to different digests in different places, which is expected behaviour that deployment tooling usually ignores.

The misleading signal

The registry's answer to "what is the digest of 1.8.2" is a present-tense fact, and the question that matters is what the digest was when each region pulled. The registry looks consistent, both deployments look identical, and the difference is in history. That is what makes this class feel impossible before someone reads the pod's imageID.

The fix

1. Deploy by digest, not by tag. The manifest references service@sha256:..., resolved once in the pipeline. This single change removes the entire class, and it also makes the deployed artefact provable, which is what an auditor asks for.
2. Make tags immutable in the registry. Most registries support it. A repush then fails loudly instead of silently changing what a tag means.
3. Treat a tag as a human label and a digest as the identity. Tags for people reading dashboards, digests for machines.
4. Wait for replication before rolling out. The pipeline should verify that the digest is resolvable in every target region before the first region deploys, which turns a mystery into a 60-second wait.
5. Record the digest in the deployment event, so the question "what was running in region B at 14:00" is answerable from a log rather than from an archaeology session.

The alert that would have caught it earlier

A check that every replica of a service is running the same digest, per environment, alerting on a mismatch outside a deployment window. It is a few lines against the orchestrator's API and it catches this, a partially completed rollout and a node that never pulled the new image.

When this is less of a concern

A single-region deployment with a pipeline that builds and deploys in one pass has little exposure, and immutable tags alone are enough there. The digest discipline becomes load-bearing when there are multiple regions, a pull-through cache, or any gap in time between build and deploy — and it is mandatory once you must prove which artefact ran, since a tag is not evidence.

What deploying by digest costs

It is not free, and the objections are reasonable. A manifest with a 71-character digest is unreadable to a human, so dashboards and rollback commands need the tag carried alongside as a label. A hotfix can no longer be shipped by repushing a tag, which is exactly the practice being removed, and teams that relied on it need a 5-minute path to build and deploy a new version instead. And a digest-pinned manifest must be generated by the pipeline rather than hand-edited, which means the pipeline becomes the only way to deploy — a benefit that is experienced as a restriction in the first month.

Choose tags only where a wrong version is harmless, such as a local development compose file. Anywhere a rollback, an audit or a second region exists, the digest is the identity. The general principle has been the same since content-addressed storage became normal in registries around 2016: a name that can be repointed is a cache key, not an identity, and using it as an identity produces failures that look impossible.