intermediate 2 min answer

An internal service-to-service certificate expires at 02:00 on a Sunday. What is the impact and what should have prevented it?

tlsautomationmonitoringoutage
Show the full answer Hide the answer

What the interviewer is testing

Whether you recognise a completely predictable failure and know why manual processes do not prevent it.

The impact

Every call to that service fails TLS validation. Depending on the callers' error handling this either produces immediate hard failures or — worse — retries that exhaust connection pools and cascade.

Detection is often slow because the error surfaces as a connection failure rather than as something recognisably certificate-related, and at 02:00 on a Sunday the on-call engineer is diagnosing a mysterious estate-wide failure rather than an obvious expiry.

Why it happened

Manual renewal depends on a person remembering, and the interval is long enough that the person has usually changed roles. This is the entire explanation, and it recurs at large and sophisticated organisations.

The certificates that cause incidents are consistently the ones outside the main path: internal service-to-service certificates, client certificates for partner integrations, certificates embedded in mobile applications for pinning, code signing certificates, and the one configured manually four years ago by someone who has left.

What should have prevented it

Automated issuance and renewal through ACME or an internal certificate authority, with automated deployment — a renewed certificate sitting unbound in a store is not renewed in any useful sense.

An inventory discovered by scanning, not by asking. The dangerous certificates are precisely the ones nobody registered.

Expiry monitoring with escalating alerts at 30, 14 and 7 days, routed to an owner, treating 7 days as an incident.

Short lifetimes deliberately. Counter-intuitively, shorter certificates are safer: they force the automation to be exercised constantly, so a broken renewal is discovered in days rather than at the next annual expiry.

Monitor the full chain, not only the leaf — an expired intermediate produces the same outage and is easier to miss.

What a strong answer adds

Monitor your monitoring. Equifax's breach went undetected for around 76 days partly because an expired certificate on a traffic inspection device meant the traffic it was meant to inspect was passing uninspected. A control whose failure is invisible provides false assurance, which is worse than no control.

Common weak answers

A calendar reminder. Renewing for three years, which makes the eventual failure less frequent and more surprising.