A service mesh runs 400 sidecar proxies with mutual TLS between services and workload certificates issued by the mesh's own authority with a 24-hour lifetime. The control plane becomes unavailable for a prolonged period. What happens, hour by hour, and what would you change in advance?

Hour by hour

Hour 0 to 1: nothing. Every sidecar holds its last configuration in memory and keeps forwarding. This is the property a well-designed data plane has - it fails static rather than failing closed - and it is why the first hour produces no alerts.

Hour 1 onwards: the system freezes rather than breaks. New pods start and get no configuration, so a deployment, a scale-out or a node replacement produces pods that cannot talk to anything. Endpoint changes do not propagate, so traffic continues to be sent to instances that have gone away, and retries mask it until they do not.

Around hour 12: certificate rotation starts failing. Sidecars normally renew well before expiry - typically at around half the lifetime - and each renewal now fails. Nothing visible happens yet, because the existing certificates are still valid.

Around hour 24: the cliff. Workload certificates expire. Mutual TLS handshakes begin failing between services whose certificates have lapsed, and because expiry times cluster around deployment times, large groups fail within the same few minutes. This is a total service-to-service outage, arriving roughly a day after the actual fault, with no gradual degradation to warn anyone.

Where it amplifies

The failure is correlated by construction: one authority, one lifetime, many workloads deployed together. It also defeats the usual recovery instinct - restarting a pod makes it worse, because a restarted pod has no configuration at all.

What the user sees

For the first hours, nothing. Then, within a few minutes, errors across every path that crosses a service boundary at once.

What stops it - mechanisms, not monitoring

1. Certificate lifetime sized against your control-plane recovery objective, not against security fashion. A 24-hour lifetime buys a 12-hour renewal window and therefore about 12 hours of tolerance. If restoring the control plane can take a day, the lifetime must be measured in days. This is the single most load-bearing number in the design.
2. Stagger expiry deliberately, by randomising the initial lifetime by a few hours, so expiry is a ramp rather than a cliff and the first failures are a warning instead of the outage.
3. Alert on the minimum remaining certificate lifetime across the fleet and on configuration age, not on control-plane health. The actionable signal is "the data plane is 40 minutes stale and the nearest expiry is in 9 hours".
4. A documented break-glass path that issues longer-lived certificates from an offline or independent authority, and that has been exercised. Under this failure the mesh's own issuance is the thing that is down.
5. Keep the control plane out of the data plane's dependency chain for steady-state traffic, which mature meshes do, and verify it with a game day that takes the control plane away for two hours.

What would have to be true for it to self-heal

Only that the control plane returns before the shortest remaining certificate expires. That is the whole of the system's resilience to this failure, and it is a single number somebody chose - which is why it belongs in the design review rather than in a default configuration file.

When this is the wrong design to run

The trade is explicit: a 24-hour lifetime limits the damage from a stolen credential to a day and costs you a 12-hour outage clock on the control plane, while a 7-day lifetime inverts both. Prefer the longer lifetime unless a compliance obligation or a real credential-theft threat model justifies the shorter one, and never choose the number from a tutorial default. For a fleet under about 20 services inside one cluster with no short-credential requirement, plain TLS with longer-lived certificates and a network policy removes this failure mode entirely, at the cost of losing the mesh's uniform identity and policy model.