On 11 December 2024 OpenAI rolled a new telemetry service across its Kubernetes fleet. Staging was clean; the change reached every cluster in under thirty minutes; all services degraded for roughly four hours. What is the failure chain, and which of its links is the one to design against?

The trigger

The new telemetry service collected Kubernetes metrics, and to do that every instance of it queried the Kubernetes API server. The published incident review states the configuration unintentionally generated resource-intensive API operations across the fleet, and that the load scaled with cluster size. A cluster with a few hundred nodes absorbed it. The largest clusters did not.

The rollout sequence is worth reading as a sequence: deployed and verified in a staging cluster on 10 December, merged at 14:23 PST the next day, applied to all clusters between 14:51 and 15:20, with impact beginning at 15:16 and running until roughly 19:38.

Why it propagated

Kubernetes' control plane is mostly independent of the data plane, and "mostly" is where this incident lives. Running pods kept running. What broke was DNS-based service discovery, which depends on the control plane to keep endpoint records current. Once the API servers were saturated, discovery stopped being updated, and services progressively lost the ability to find each other. The workloads were healthy and unreachable.

The second link is the one that made it long: remediation required the control plane that was saturated. Engineers needed API access to remove the offending service, and the API servers had no capacity to serve that request. This is the locked-out pattern, and it is what turns a twenty-minute mistake into a four-hour outage.

Why detection lagged

DNS caching. Cached records kept resolution working after the control plane stopped updating them, so the fleet looked healthy for the window in which the rollout could have been stopped cheaply. The staged rollout reached 100% before the first symptom appeared. A cache that smooths over a dependency's failure is also a cache that hides it long enough for the blast radius to complete.

The structural fix versus the tempting local fix

The tempting fix is to make the telemetry agent cheaper: fewer queries, a watch instead of a poll, a rate limit in the client. Worth doing, and it does not address the shape of the failure.

The structural fixes are three:

1. Treat control-plane load as a first-class resource with enforced limits, so no client, telemetry included, can consume the API server's capacity. Priority and fairness controls exist for exactly this and are usually left at defaults.
2. Break-glass access that does not depend on the failing component. The incident review names this directly. If the only route to fix the control plane is through the control plane, the recovery time is unbounded.
3. Stage rollouts by blast radius, not by clock. Staging then everything is two stages. The missing stage is one large production cluster, held long enough for the slowest detection path — which here was DNS TTL, not metrics — to report.

The general lesson

Monitoring software is production software with fleet-wide reach. It is deployed everywhere by definition, it usually talks to the platform's control plane, and it is frequently exempt from the progressive-delivery rules applied to product code because it is "just telemetry". Two of the largest publicly documented control-plane outages of recent years started in an observability component. Ask of any fleet-wide agent: what does it query, how does that scale with cluster size, and what happens when 100% of instances do it at once?

When this is over-caution

A cluster of 30 nodes with one API server client per node does not need a staged rollout per cluster: the load is trivial and the ceremony costs more than the risk. The threshold worth using is whether any single cluster's control plane would notice the agent at 100% deployment, which is a load test somebody can run in an afternoon rather than a policy debate.