A Kubernetes upgrade to 1.24 removes a deprecated node label. Within minutes the cluster's pod network loses all routes and stays down for over five hours, and the configuration responsible is not in any repository. What kind of dependency is this, and how would you have found it beforehand?

The trigger

Reddit's Pi Day outage on 14 March 2023 ran 314 minutes. The mechanism: their Calico installation selected route reflectors by a node label, node-role.kubernetes.io/master, and Kubernetes 1.24 removed that label in favour of control-plane. As the upgrade progressed, the selector matched nothing. Routes for the first upgraded node dropped, which was expected; routes for every node then dropped, which was not.

Why it propagated

A route reflector is a control-plane component for the network, distributing BGP routes so every node does not peer with every other. When the selector stopped matching, the reflectors ceased to be reflectors, and the mesh they were summarising had never been configured to exist without them. The failure was total rather than partial because the dependency was structural, not incremental.

The organisational half of the trigger matters as much as the technical half. The reflector configuration had been applied years earlier through a third-party tool by a team that no longer existed, was in no repository, and keyed on a label the platform provided. Nobody could have read the 1.24 release notes and connected them to this cluster, because the affected configuration was not written down anywhere a reader would look.

Why detection lagged

The metrics died with the cluster. Logs survived because they were deliberately shipped by a path that did not depend on Kubernetes, which is the one design decision that helped, and there were billions of lines of them to search under pressure.

The structural fix versus the tempting local fix

The local fix is "read release notes more carefully", which fails the next time for the same reason it failed this time: the dependency was invisible to the person reading them.

The structural fixes:

• Inventory the out-of-band configuration. Not "is everything in Git", which is aspirational, but a specific question: what in this cluster was applied by hand or by a tool that is no longer used, and what does it select on? That inventory is a day of work and it is the actual artefact this incident is missing.
• Treat platform-provided labels and annotations as an API surface with deprecation consequences. Anything selecting on them is a consumer, and a cluster upgrade is a breaking change to that API.
• Upgrade a full-shaped non-production cluster first, with the same add-ons, the same network plugin and the same hand-applied history. A clean test cluster proves the upgrade works on a cluster nobody has been living in for four years.
• Keep at least one telemetry path independent of the cluster, which Reddit had for logs and not for metrics.

The general lesson

The risky part of an upgrade is not the software, it is the accumulated configuration the software is being upgraded underneath. Every long-lived cluster carries settings whose authors have left, whose purpose is undocumented and whose dependencies are implicit. A dependency that exists only as a selector string is invisible to code search, to release notes and to the person approving the change.

When this is the wrong lesson to draw

The conclusion "never upgrade" or "always run clusters for a year behind" costs more than it saves: unsupported versions accumulate unpatched vulnerabilities and the eventual jump is worse. The lesson is inventory and blast radius, not delay.