On 14 March 2023 a Kubernetes upgrade at Reddit caused an outage of about 314 minutes. The upgrade removed a node label that an internal network component had been selecting on, and that component's configuration had been hand-edited through a CLI and committed nowhere. Where do you start investigating, and what is the diagnosis?

The first three things to look at, in order

1. What changed in the upgrade's own release notes about identifiers. Kubernetes 1.24 replaced the node-role.kubernetes.io/master label with control-plane. A renamed label is a breaking change to anything selecting on it, and nothing in the cluster reports that a selector has stopped matching - it simply matches zero objects.
2. Which components select on labels, and where those selectors are defined. In Reddit's case the network plugin's route reflectors were designated by that label; with the label gone, the routing mesh lost its reflectors and pod networking degraded.
3. Whether each component's configuration exists in version control. This is the question that turns a two-hour incident into a five-hour one.

The diagnosis

The failure was a selector that matched nothing; the outage length was caused by the configuration being committed nowhere.

The route-reflector settings had been fetched with the plugin's own CLI, hand-edited and pushed back, by engineers who had since left. There was no file to read, no history to diff, and no declaration to compare against reality. So the engineers debugging it could not answer the first question any responsible would ask - "what is this supposed to look like?" - from anything other than the failing cluster itself.

Two further amplifiers: Kubernetes has no downgrade path, so reverting meant restoring from backup using a procedure written for older software and never exercised; and the cluster's own metrics died with it, leaving logs as the only signal.

The misleading signal

The upgrade completed successfully. Every node reported ready. The control plane's own health checks pass in this failure mode, because nothing is unhealthy - a set of objects is simply empty, which is a valid state. An investigation that trusts the upgrade's success indicator will look for a coincidental cause.

The fix, structurally

• Every piece of cluster configuration in version control, without exception. The GitOps claim that matters is not "we deploy from Git"; it is "there is no state in the cluster that was not declared somewhere readable." Anything edited through a CLI and pushed back is state with no source.
• Reconciliation that reports drift, so hand edits are visible rather than merely discouraged.
• A test cluster upgraded first, with the same configuration. This defect would have appeared there in minutes.
• An audit of selectors against deprecation notes as a standing upgrade step. Label and API renames are the most common form of this failure.
• A rehearsed restore. A backup procedure first executed during an outage is a plan, not a capability.

The alert that would have caught it earlier

Alert on selectors that match zero objects, for components where zero is never correct - route reflectors, ingress backends, monitoring targets. It is a cheap check and it catches an entire class of rename-driven failure that nothing else surfaces.

When this is the wrong lesson

Not every hand edit is a defect. Emergency changes made during an incident are legitimate and should be possible - the requirement is that they are reconciled or committed afterwards, not that they are prevented. A GitOps setup that reverts an engineer's emergency fix sixty seconds after they make it is correct in steady state and needs a documented pause mechanism, or people will disable the controller instead.