A payments gateway of Razorpay's kind authenticates 180 bank and merchant partners with mutual TLS. One partner's client certificate expires at 00:00 UTC on a public holiday. What happens, and what design would have prevented the call at 00:04?

What happens, minute by minute

• 00:00 - the certificate is past its notAfter. Every new TLS handshake from that partner fails. Existing connections may survive, since expiry is checked at handshake, so the failure looks intermittent and depends on connection pooling and keep-alive timeouts.
• 00:01 - the partner's error rate goes to 100% for new connections. Your side logs a TLS handshake failure, which most observability stacks bucket as a network error rather than an authentication one, so the dashboard shows "connection errors" and not "expired certificate".
• 00:04 - someone is paged, on a holiday, for a problem that can only be fixed by a person at the partner who is also on holiday, and who needs their own certificate authority process to issue a replacement.
• The rest of the day - a manual workaround is requested. The dangerous ones get accepted: disable verification for that partner, extend trust to a self-signed certificate, or allow expired certificates temporarily. Each is a permanent security change made under time pressure, and each tends to outlive the incident.

Where it amplifies

If the partner is a payment rail, every transaction routed to it fails and the retry logic sends them again, so a certificate expiry becomes a queue backlog and a reconciliation exercise. And because mutual TLS is symmetric, the same failure happens in the other direction when your own certificate expires - and then it happens to 180 partners at once.

What stops it

• An overlap window, by design. Trust both the old and the new certificate for a period - weeks, not hours - so rotation is a change either side can make independently. For mutual TLS this usually means accepting two client certificates per partner during the window, or trusting an intermediate authority rather than pinning a leaf.
• Trust the issuer, not the leaf. Pinning a specific certificate makes every renewal a coordinated change; trusting a partner's intermediate certificate authority with a name constraint makes renewal their internal matter.
• Expiry as inventory, not as a surprise. A register of every partner certificate with its expiry date, alerting at 60, 30 and 7 days, and a named owner on each side. With 180 partners and roughly annual certificates, a certificate expires somewhere in the estate every other day - this is a continuous process, not an event.
• Classify the error properly. A TLS handshake failure should raise a distinct alert from a connection failure, with the certificate subject and expiry in the message. That alone turns a 30-minute diagnosis into a 30-second one.
• Rehearse rotation, so the first time anyone performs it is not during an incident.

What would have to be true for it to self-heal

Automated issuance and renewal on both sides, with short-lived certificates rotated often enough that the mechanism is exercised constantly. The counter-intuitive result holds here: certificates that live for days are far safer than certificates that live for a year, because the renewal path is used daily and breaks loudly in office hours rather than silently at midnight on a holiday. That is achievable inside your own estate and much harder across 180 organisations, which is why the overlap window is the pragmatic answer at the partner boundary.

When this is the wrong level of effort

For a handful of partners whose integration is low-value and low-volume, a calendar reminder and a documented runbook is proportionate. The inventory-and-overlap machinery earns its place when the partner count means expiries are weekly - and the threshold is lower than people expect, because the cost of the incident does not scale down with the partner's size.