At 03:12 on a Sunday about 6% of internal service calls begin failing TLS handshakes with an unknown-authority error. The failure rate climbs over four hours to 40% and then stops climbing. No deployment happened. A certificate authority rotation was completed on Friday. What happened, and in which order should it have been done?

The first three things I would look at, and why in that order

1. The issuer of the certificates being rejected, from a failing client's perspective. The error names the problem class - the presenter's chain is not anchored in the verifier's trust store - and the issuer field says which authority signed it.
2. Whether the failure rate tracks certificate renewal. A rate that climbs smoothly and then plateaus is a population converging, not a component failing. That shape is the diagnosis.
3. Which trust bundle each workload actually holds, by version, rather than which one the configuration management system believes it pushed.

The diagnosis

The rotation switched the issuing authority. From Friday onward, every workload that renewed its leaf certificate received one signed by the new authority. Meanwhile, some verifiers still held only the old root in their trust store - because the bundle update was packaged with a deployment, a configuration refresh or a pod restart that had not yet reached them.

So the failure population is the cross product of "presenter has already renewed" and "verifier has not yet updated". With a 24-hour leaf lifetime and renewal at about half of that, the presenter population converges within roughly 12 to 24 hours - which is exactly the shape observed: a four-hour climb to 40%, then a plateau once the renewals finished, leaving a stable fraction of stale verifiers failing.

The rotation was correct in content and wrong in order.

The order it should have been done in

1. Distribute the new root and intermediate to every verifier, trusting old and new simultaneously. Nothing changes behaviourally; this step is purely additive and is therefore safe to take slowly.
2. Verify coverage before proceeding. Export a per-workload trust-bundle version as a metric and confirm 100% of the fleet reports the new bundle. This gate is the whole control, and it is the step that was missing.
3. Switch issuance to the new authority. Leaves now chain to a root everyone already trusts, so the change is invisible.
4. Wait out the longest leaf lifetime - with 24-hour certificates, at least 48 hours of margin - so no certificate signed by the old authority remains in use.
5. Remove the old root from trust stores, last, and only after confirming no certificate in the fleet still chains to it.

The rule generalises: trust is distributed before it is used, and withdrawn after it stops being used. The same ordering applies to JWT signing keys published through a JWKS endpoint, to SSH certificate authorities, and to code-signing roots. Reversing it produces an outage whose size is set by how fast the population converges, which is why it tends to arrive hours after the change and nobody connects the two.

The misleading signal

"No deployment happened", which is true and is why the Friday change was not suspected. Certificate renewal is a background process that makes changes without a deployment, so a PKI change has a delayed, self-propagating blast radius - the most confusing failure shape there is.

The alert that would have caught it earlier

Two signals: handshake failures broken down by presented issuer, and the minimum trust-bundle version across the fleet. The first turns "TLS errors" into "clients that trust only the old root are rejecting the new issuer", which is the whole answer; the second would have blocked step 3 before it was taken.

Common weak answers

• "Roll back the rotation." It does not help: certificates already issued by the new authority stay issued, and rolling back issuance leaves them unverifiable for longer.
• "Restart everything to pick up the new bundle." It will work, and it is a fleet-wide restart during an outage because a coverage gate was skipped - and it hides the stragglers that caused it, which are typically containers with a trust store baked into the image or sidecars that read the store only at start-up.
• "Push the bundle again." The push was recorded as successful; prefer a per-workload reported version over a configuration system's own success record, because the two disagree exactly where it matters.
• "It is clock skew." A reasonable alternative hypothesis, and distinguishable: skew produces not-yet-valid errors rather than unknown-authority ones.

What the dual-trust window costs

Trusting old and new simultaneously means two authorities can issue certificates your fleet accepts, so the old key must stay protected until it is removed. Choose a window long enough to find the stragglers and short enough that nobody forgets step 5 - for 24-hour leaves, 48 to 72 hours is usually right. The exception is a key compromise, where the order inverts: remove the compromised root immediately and accept the outage, because continuing to trust it is worse.