Your pipeline promotes one container digest from staging to production, and the team says production therefore runs exactly what was tested. The deployment chart values, the feature-flag defaults and the migration set are each resolved fresh at deploy time. Why is the promotion claim false, and what is the smallest change that makes it true?
Show the full answer Hide the answer
The mechanism
A running service's behaviour is not a function of its binary. It is a function of four inputs: the artefact, the configuration it is started with, the schema it reads, and the flag values it evaluates. Promoting the digest pins one of the four and leaves three free.
That is why the claim fails. Staging exercised the digest paired with staging's values. Production pairs the same digest with a different set. If staging sets a cache TTL of 60 seconds and production sets 3,600, a defect that only appears when entries live past five minutes was never reachable in staging, no matter how many times the artefact was promoted.
What the promotion actually proved
Promotion proves the artefact was not rebuilt, which rules out a real and common failure: a second build of the same commit picking up a different base image or a floating dependency. That is worth having.
It does not prove the release was tested, because the release is the pair. A typical chart resolves on the order of a hundred values across environment files, platform defaults and injected secrets, and nothing in a digest-only promotion records which of those were in force when the tests passed.
The smallest change that makes the claim true
Make the promoted thing a (digest, configuration revision) pair, and record the pair in the deployment record:
- Keep the environment's values in version control and resolve them at build time into a named configuration revision, not at deploy time.
- Have the pipeline promote the pair, and have the deploy refuse to run if the pair it was handed is not the pair that passed the previous stage.
- Treat flag defaults the same way. A flag whose default differs between environments is configuration wearing a different hat.
- Put the schema version in the pair too, so "which migration set was applied when this passed" has an answer.
The rule worth remembering: the release unit is the smallest set of inputs that, held constant, makes behaviour reproducible. Anything outside that set is an untested variable, however carefully the artefact was promoted. Prefer the pair unless you can show the extra inputs cannot change behaviour.
When this is the wrong answer
If the only differences between environments are endpoint strings and secrets the platform injects, and every behavioural setting is identical, then a digest plus a values file committed in the same repository at the same commit is already a pair. Building a separate bundle artefact for that case costs a pipeline stage and the ongoing overhead of a second artefact store, and buys nothing.
The test is specific: list every configuration key that differs between staging and production, and ask which of them can change behaviour rather than destination. If that list is empty, stop here.
Common weak answers
- "Use the same configuration everywhere." Appealing and impossible: endpoints, capacity, quotas and secrets must differ. The goal is to pin which values were in force, not to abolish difference.
- "Add more staging tests." More tests against the wrong pair still test the wrong pair.
- "Deploy by tag instead." This moves in the wrong direction, and it is a separate failure from the one being diagnosed here.