On 11 December 2024 OpenAI rolled a new telemetry service out across every Kubernetes cluster. Within about half an hour the API servers were saturated and services could no longer resolve one another; full recovery took until the evening. What turned a monitoring change into a total outage, and which of the contributing factors would you fix first?

What happened

OpenAI's published postmortem describes a new telemetry service intended to improve Kubernetes observability. It was deployed to staging on 10 December and behaved normally. The change merged on 11 December at 14:23 PST and was applied across all clusters between 14:51 and 15:20; impact ran from 15:16 to 19:38 PST.

The service issued Kubernetes API calls whose cost scaled with the size of the cluster. In staging and in small clusters the load was unremarkable. In the largest production clusters the same code saturated the API servers, and once the control plane was degraded, DNS-based service discovery stopped being updated and services lost the ability to find each other.

Why the blast radius was every cluster at once

The rollout was staged by environment, not by the dimension the failure scaled with. Staging proved the change was correct; it could not prove it was safe, because the property that broke — request cost as a function of object count — was absent from staging by construction.

A change whose cost scales with cluster size must be rolled out in order of cluster size, with a soak on the largest one before the fleet. Environment-based promotion is an assurance that the code runs, not an assurance that it runs at your biggest number.

Why detection lagged the rollout

DNS resolution results were cached. Workloads kept resolving each other from cache after the control plane had already stopped serving correctly, so the fleet looked healthy while the change continued to propagate. By the time the caches expired and calls began failing, the change was everywhere.

This is the general shape: any cache between a broken control plane and the workload converts a fast, obvious failure into a slow, fleet-wide one. The cache is doing its job. It is also destroying the signal that rollout automation needs to halt.

Why recovery was slow

Removing the offending service required the Kubernetes API — the thing that was saturated. The tool needed to fix the problem was inside the problem. This lockout, not the defect, is what turned minutes into hours.

The fix order

1. Break the lockout. Reserve control-plane capacity for administrative traffic so operators can always act — Kubernetes API Priority and Fairness exists for exactly this, and a documented break-glass path that does not depend on the degraded component is the single highest-value change, because it divides the duration of every future control-plane incident.
2. Stage by the dimension that scales. Largest cluster first, with an explicit halt condition on control-plane latency and inflight request counts, not on workload health.
3. Treat the cache as a detection hazard. Rollout gates should read a signal that is not cached — the API server's own latency and rejection metrics — rather than inferring health from workloads that may be coasting.

When this is the wrong lesson to draw

The tempting conclusion is "do not run telemetry agents against the control plane". That is over-correction: cluster-state telemetry is how you know what is running. The defect was not observing the control plane; it was issuing unbounded, size-proportional queries against it and shipping that everywhere in twenty-nine minutes. A small team with three clusters of forty nodes will never hit this; the failure is specific to fleets where one cluster is an order of magnitude larger than the one you tested on.