Certificate Rotation Overlap

A payments gateway authenticates 180 partners with mutual TLS. One partner's client certificate expires at 00:00 UTC on a public holiday. Every new handshake from that partner fails; existing pooled connections survive until they are recycled, so the failure looks intermittent. Your logs record a TLS handshake error, which most observability stacks bucket as a network problem, so the dashboard says "connection errors" rather than "expired certificate".

By 00:04 someone is paged for a problem that can only be fixed by a person at the partner, who is also on holiday, and who needs their own certificate authority process to issue a replacement. The dangerous outcomes are the workarounds: disable verification for that partner, trust a self-signed certificate, allow expired certificates temporarily. Each is a permanent security change made under time pressure, and each tends to outlive the incident.

The structural fix is not a better reminder. It is to make rotation a change either side can make alone, by trusting old and new simultaneously for a window measured in weeks.

Why it matters

Mutual TLS is symmetric, so this failure has two directions. A partner's expiry affects one integration; your own expiry affects all 180 at once. The second case is the one that makes a rotation overlap a reliability requirement rather than a convenience.

The arithmetic also makes it continuous rather than occasional. With 180 partners on roughly annual certificates, something in the estate expires every other day. That is not an event to handle, it is a process to run - and processes need inventory, ownership and alerting, not a calendar entry.

Implementation patterns

• Trust an intermediate authority with name constraints rather than pinning a leaf certificate. Renewal then becomes the partner's internal matter and needs no change on your side at all. This is the single highest-value move available.
• Where pinning is required, accept two certificates per partner during the window, with the overlap measured in weeks. Expire the old entry on a scheduled job, not by hand.
• Keep a certificate inventory - subject, issuer, expiry, partner, owner on both sides - and alert at 60, 30 and 7 days. Derive it from what is actually in the trust store rather than from a spreadsheet.
• Classify handshake failures distinctly, with the certificate subject and expiry in the alert text. This alone turns a 30-minute diagnosis into a 30-second one.
• Rotate your own credentials often and automatically. Short-lived certificates are safer than annual ones because the renewal path is exercised daily and fails loudly in office hours - the opposite of the intuition most teams start with.
• Rehearse a rotation, so that the first time anyone performs the procedure is not during an incident.

Industry example

The industry has moved decisively in this direction. Public web certificates have gone from multi-year to 90-day lifetimes with automated issuance since Let's Encrypt made it practical in 2015, and service meshes issue workload certificates lasting hours with rotation handled by the infrastructure. Both are the same bet: a renewal that happens constantly is a renewal that works.

Partner boundaries are where that bet is hardest to place, because automation requires agreement between organisations. That is exactly why the overlap window is the pragmatic answer at the boundary: it does not require the partner to automate anything, only that both sides may change independently.

Failure scenarios

• Midnight expiry on a holiday, with no one available on either side and only unsafe workarounds on the table.
• Your own certificate expiring, breaking every partner at once.
• Handshake failures misclassified as network errors, so the diagnosis takes the length of the outage rather than a minute.
• A workaround that persists - verification disabled for one partner, found during an audit two years later.
• An intermediate certificate expiring, which breaks every partner whose chain includes it and is tracked by nobody because it is not a leaf.
• Renewal performed but not deployed, where the new certificate exists in the authority and the running service still presents the old one - which is why the inventory must read the live trust store.

Trade-offs

An overlap window means two valid credentials exist at once, which doubles the surface for that partner during the window: a compromised old certificate is still accepted until the overlap closes. That is the honest security cost, and it is bounded by the window length and by whether the expiry job actually runs.

When not to use it

Where the security model requires that exactly one credential is valid at any moment - a high-value settlement rail, a regulator-mandated control - an overlap may not be permitted. The answer there is a scheduled coordinated rotation with both parties present, treated as a planned change with a rollback, which is expensive and occasionally correct.

And for a handful of low-volume partners, a calendar reminder and a documented runbook is proportionate; the inventory-and-overlap machinery earns its place when the partner count makes expiries weekly. The threshold is lower than teams expect, because the cost of the incident does not scale down with the partner's size - a small partner's expiry still pages someone at midnight.

Interview question

Q: You run mutual TLS with 180 partners. Design the certificate lifecycle so that no single expiry can cause an incident, and tell me what you would monitor.

What a strong answer covers: trusting the partner's intermediate authority with name constraints rather than pinning leaves; an overlap window of weeks where pinning is required, with an automated job to close it; an inventory derived from the live trust store with owners on both sides and alerts at 60, 30 and 7 days; distinct classification of handshake failures with subject and expiry in the alert; the symmetry of mutual TLS and the all-partners blast radius of your own expiry; intermediate-certificate expiry as the dependency nobody tracks; and the security cost of the overlap stated plainly.

Quick check

Quiz: Why does an expired client certificate produce intermittent rather than total failure at first? Because expiry is enforced at the TLS handshake, so pooled connections established before expiry keep working until they are recycled, while every new connection fails.

Flashcard: What does trusting a partner's intermediate authority buy over pinning their leaf certificate? Renewal becomes entirely the partner's internal matter with no change on your side - at the cost of accepting any leaf that authority issues, which is why name constraints are part of the pattern.