On 14 March 2023 Reddit upgraded a Kubernetes cluster from 1.23 to 1.24 after the same upgrade had already succeeded elsewhere, and was down for 314 minutes. The upgrade removed a node label that a hand-configured Calico route reflector selected on. Which parity assumption failed?

The trigger

Kubernetes renamed control-plane node terminology in the 1.20 series and removed the older node-role.kubernetes.io/master label in 1.24, including from running clusters. Reddit's route reflectors — the BGP relays Calico uses so that every node does not peer with every other node — selected their servers using that label. When the label went, the selector matched nothing, the relays disappeared, and pod networking stopped resolving across the cluster.

Two details make it a parity story rather than a Kubernetes story. Only the largest and oldest clusters used route reflectors at all, and the configuration had been made by hand years earlier by people who had since left, and was committed nowhere.

Why the rehearsal did not catch it

The upgrade had been performed successfully on other clusters. Those clusters were topologically similar and had a different provenance: built later, by newer tooling, without the hand-edit.

That is the transferable point. Parity is usually measured as similarity of shape — same versions, same topology, same manifests — when the property that matters is similarity of history. A cluster created last year is not a rehearsal for a cluster hand-tuned in 2019, because the difference between them is exactly the undeclared state that no manifest describes.

Why rollback was not available

Kubernetes has no supported downgrade. The recovery path was a restore from backup, using a procedure written for older software and never exercised in production, which is how a networking misconfiguration became a five-hour outage rather than a fifteen-minute one. The blast radius of an upgrade is set by the credibility of its reverse, and an untested restore is not a reverse.

The structural fix versus the tempting local fix

The tempting fix is to restore the label or re-point the selector. That is necessary and it addresses one instance.

The structural fixes are:

1. Drift detection that answers "what exists here that nothing declares?" — a reconciliation report per cluster, not just a plan-and-apply that only looks at declared resources.
2. A per-environment deviation register, kept small on purpose, so the question "how is this cluster different?" has an answer that is not archaeology.
3. Rehearse on a clone of the specific environment, or accept that the rehearsal only covers the environments that share provenance with the one rehearsed.
4. Make the reverse real before the forward: for a one-way upgrade, that means a tested restore with a measured time, and a decision in advance about when you stop debugging and start restoring.

Common weak answers

• "They should have tested in staging." They tested on other production clusters, which is stronger than staging, and it still did not cover the difference that mattered.
• "Pin the Kubernetes version." Deferring an upgrade makes the eventual one larger and its undeclared state older.
• "Use GitOps." Correct direction and insufficient as stated: a declarative tool applies what is declared and is silent about hand-made state, which is precisely what this incident was.