Provenance Parity

Parity is normally measured as similarity of shape: same versions, same topology, same manifests, same instance types. That is the dimension teams can see, so it is the dimension they buy.

The dimension that decides whether a rehearsal transfers is provenance: how each environment came to exist and what has been done to it since. Two clusters can be identical in every declared respect and differ entirely in undeclared state, because one was hand-tuned three years ago and the other was created last quarter by newer tooling.

Why it matters

A successful rehearsal is the main evidence used to authorise a risky change. If the rehearsal environment has a different history, that evidence does not cover the difference, and the difference is precisely the undeclared state nobody can enumerate.

This is also why "we tested it in staging" is weaker than it sounds and why "we did this on three other production clusters" can be weaker still: the clusters that upgraded successfully may be the ones that were never hand-edited.

Implementation patterns

• Drift detection that answers the open question. Not "does the declared state match?" but "what exists here that nothing declares?" — a reconciliation report over actual resources, since a plan-and-apply run only inspects what is in the manifests.
• A deviation register per environment, deliberately short, listing every known difference and its owner. A register with 200 entries is a to-do list; one with 5 is a control.
• Rebuild rather than upgrade where the cost allows: a replacement environment built by current tooling has no history, which is the cheapest way to buy provenance parity.
• Rehearse on a clone of the specific environment, restored from its own state, rather than on a representative one.
• Date the environment. Recording when and by what an environment was created makes the question askable at all.

Industry example

Reddit's Pi-Day outage on 14 March 2023: a Kubernetes upgrade from 1.23 to 1.24 removed the node-role.kubernetes.io/master label, which a Calico route reflector configuration used to select its peers. Only the largest and oldest clusters used route reflectors, the configuration had been made by hand years earlier by people who had left, and it was committed nowhere. The same upgrade had already succeeded on clusters that never carried that configuration. The outage ran 314 minutes, and because Kubernetes has no downgrade, recovery meant a restore procedure that had not been exercised.

Failure scenarios

• The successful rehearsal that proves nothing, because it ran on a newer environment.
• Undeclared state discovered during an upgrade, when the people who created it have left.
• Rollback that is not a rollback — a one-way upgrade whose reverse is an untested restore.
• Configuration that lives only in a running system, so no search of the repositories finds it.
• Convergent drift, where hotfixes applied to production during incidents were never applied to the rehearsal environment.

Trade-offs

Provenance parity is expensive in a way shape parity is not: it pushes towards rebuilding environments rather than maintaining them, which costs migration work and can break long-lived local state. The payoff is that rehearsals transfer and upgrades stop being archaeology. A reasonable middle is to rebuild the environments whose failure would be catastrophic and accept drift on the rest, with the deviation register making the choice explicit rather than accidental.

When not to use it

For a single ephemeral environment recreated from code on every pipeline run, none of this applies — it has no history to diverge, and the useful parity questions are about data volume and traffic shape instead. The concept earns its keep only for long-lived environments, which is where the expensive incidents happen.

Interview question

Q: Your team is about to perform a one-way upgrade on the oldest production cluster. It has already succeeded on three newer ones. What do you check before approving it, and what is your stopping rule if it goes wrong?

What a strong answer covers: asking how the target differs in history rather than in shape · a drift report for undeclared resources · a check for hand-made configuration and for selectors depending on platform labels · timing the restore path in advance so the reverse is credible · and a pre-agreed stopping rule with a clock, because the decision to abandon debugging and restore is the one nobody makes well at hour three.

Quick check

Quiz: Why can a rehearsal on an identical-looking environment fail to transfer? Because parity of shape does not imply parity of history, and undeclared state is exactly what no manifest shows.

Flashcard: What question does real drift detection ask? — What exists here that nothing declares, rather than whether declared state matches.