Machine identity expiry  / field guide
Practitioner field guide · 29 August 2026

The renewal failed a month before anyone noticed

Every engineer knows certificates expire. Organisations that know it perfectly well still lose whole days to it, and the reason is never the expiry date. Reconstructed from ten published incidents at Bazel, DigiCert, Microsoft, Let's Encrypt and Ericsson, from the ballots and RFCs that are cutting public certificate lifetimes to 47 days, and from the issue trackers of the tools that renew them. What you get: the one number that predicts whether an expiry becomes an outage, and where to put it on a dashboard.

22 primary sources 17 distinct hosts 10 incidents 6 figures Evidence through August 2026 Read: 44 min
01

The territory

The problem, stated without naming the technology: one system stops trusting another at a moment decided months earlier, by a process nobody is watching, on behalf of an owner who may have left.

2.6h
Average time to identify a certificate-related outage, before any time spent fixing it
30 days
The Bazel renewal failed silently for about a month before the certificate expired
47
Maximum days of public certificate validity from 15 March 2029, down from 398
20% → 2.9%
Renewal lapse rate before Let's Encrypt, against lapses among its own renewals

On 26 December 2025 at 07:35:39 UTC, a certificate covering *.bazel.build expired and continuous integration builds around the world began failing. The interesting part of the Bazel team's postmortem is not the expiry. It is the two dates on either side of it. On 30 October, someone removed the DNS records for an unused subdomain, docs-staging.bazel.build, which was still listed on the certificate. Around 26 November the managed renewal process ran, found that a listed domain was no longer reachable, and failed. It then failed again, repeatedly, for a month. In the postmortem's own words: "These failures never triggered any notifications."

Then the detection gap. The certificate expired at 07:35 and users reported broken builds on Slack and GitHub by 09:00. A team member in Munich saw the GitHub issue at 12:15 and was not in a position to act. A team member in New York began investigating at 15:08. The replacement certificate finished propagating at 20:31. The renewal took minutes. The thirteen hours were spent on everything except renewing.

The finding worth your Thursday

Certificate lifetime is not the number that governs this failure. The number that governs it is the gap between when you renew and when the credential dies, expressed as the count of consecutive silent renewal failures you can absorb. Call it the slack. Three independent sources, spanning thirteen years, converge on the same ratio without ever putting it that way: the 2012 Stanford and CMU paper that started the short-lived certificate movement suggests fetching daily against a four-day lifetime; RFC 8739 requires that the next certificate be available "halfway through the lifetime of the currently active certificate at the latest"; and Let's Encrypt tells operators of its six-day certificates to renew "every two to three days". That is a slack of two to four attempts, every time. Most enterprise estates run a slack of one, because a managed renewal fires once and nobody counts the attempts.

This matters more each year because the lifetime is being taken away from you. In April 2025 the CA/Browser Forum passed ballot SC-081v3 with 25 issuer votes in favour and none against, cutting the maximum validity of a public TLS certificate from 398 days to 47 on a fixed schedule: 200 days from 15 March 2026, 100 days from 15 March 2027, and 47 days from 15 March 2029. The 200-day tier has been in force for five months as this guide is written. The ballot's stated reason is staleness rather than automation: "Certificates are representations of a point in time state of reality... The more time passes from that moment of issuance, the more likely it becomes that data represented in the certificate diverge from reality." Chrome's root program is more direct about the intent. Reducing lifetime "encourages automation and the adoption of practices that will drive the ecosystem away from baroque, time-consuming, and error-prone issuance processes."

Figure 1 · Thirteen years of taking the manual option away

2012
paper proposes
4 days

2015
Let's Encrypt
90 days

2020
capped at
398 days

2025
ballot passes
25 to 0

2026
200 days

2029
47 days

2012
paper proposes
4 days

2015
Let's Encrypt
90 days

2020
capped at
398 days

2025
ballot passes
25 to 0

2026
200 days

2029
47 days

Each step down was argued as a security measure and functions as a deadline for automation. Sources: Topalovic et al., W2SP 2012, Aas et al., CCS '19, ballot SC-081v3, Let's Encrypt, 2025.
Diagram source

Who has published on this, and what they run

The public record here is lopsided in a useful way. Certificate authorities are required by root-store policy to file public incident reports, so the CA side of this problem has a corpus of unusually candid, technically detailed postmortems in Mozilla's Bugzilla that almost no architect reads. The subscriber side is thinner and mostly consists of small organisations being honest, of which the Bazel report is the best recent example. The hyperscalers publish when the failure is theirs: Microsoft's post-incident review of the December 2025 Azure Government and Azure China outage is a first-party account of an automated key rotation causing the outage rather than preventing one. Uber and Netflix have published on internal machine identity at scale. Ericsson, whose expired certificate cut off roughly 32 million UK mobile subscribers in December 2018, has published nothing that is still reachable, which is itself worth knowing.

What this guide does not cover

Not covered: cryptographic algorithm choice and the post-quantum migration; certificate transparency and misissuance detection; human and workforce identity, including SSO and session lifetime; secret scanning and leaked-credential response; and the compliance paperwork around key ceremonies. The scope is the operational lifecycle of a machine credential: how it is obtained, how it is refreshed, what watches the refresh, and what happens when the refresh fails or is forced.

02

Four planes, and the one everybody skips

Every credential lifecycle in the published systems decomposes the same way: something issues, something distributes, something consumes, and something is supposed to observe. The fourth plane is where every incident in section 4 actually happened.

Start with the arithmetic, because it is simple and almost nobody writes it down. A credential has a lifetime L. A renewal process attempts to replace it starting at some point, leaving a window W between the first attempt and expiry. The process retries on some interval R. The number of consecutive failures you can survive without a human is W divided by R. That is the slack, and it is the whole game. Bazel's slack was effectively one: a managed renewal that began about thirty days out and then latched into a failing state with no alert, so a single unresolved failure consumed the entire window. cert-manager's default is more generous, renewing "2/3 through its actual duration", which puts W at a third of the lifetime, with retries backing off from one hour to a cap of thirty-two hours. On a 90-day certificate that is thirty days of window and roughly twenty attempts. On a 47-day certificate it is fifteen days and roughly ten. On a six-day certificate it is two days, which is why Let's Encrypt tells you to renew every two to three days and check the renewal API daily rather than relying on backoff.

Figure 2 · The slack, which is the only number that predicts the outage

Yes

No. Retry after R.

No.
W / R left

Yes

Credential issued
lifetime L

Renewal begins
window W opens

Attempt
succeeds?

Renewed.
Window closes.

Window
exhausted?

Expiry.
Outage begins.

Yes

No. Retry after R.

No.
W / R left

Yes

Credential issued
lifetime L

Renewal begins
window W opens

Attempt
succeeds?

Renewed.
Window closes.

Window
exhausted?

Expiry.
Outage begins.

The window between first renewal attempt and expiry, divided by the retry interval, is the number of silent failures you survive. The 2012 paper states the principle directly: a failed fetch is harmless because the previous credential is "active for a few more days giving the administrator and the CA ample time to fix the problem". Topalovic et al., cert-manager defaults.
Diagram source

The second thing to internalise is that shortening the lifetime does not by itself shorten the slack, and lengthening it does not lengthen the slack. They are independent knobs and most teams only touch one. An organisation moving from 398-day to 47-day certificates while keeping a renewal process that fires once, thirty days out, has not reduced its slack at all; it has multiplied the number of times per year that its single attempt has to work, from roughly once to roughly eight times. If the attempt succeeds 99% of the time, the annual probability of at least one expiry goes from about 1% to about 8% for that one certificate. Multiply by the estate. This is reconstruction rather than report: no source computes it, but it follows directly from the ballot schedule and any fixed per-attempt failure rate.

Figure 3 · Reference architecture: four planes, reconstructed across the published systems

Observation plane

Consumption plane

Distribution plane

Issuance plane

Public CA via ACME
or internal CA

Profile and policy
lifetime, SANs, key type

Renewal agent
certbot, cert-manager, SPIRE

Secret store
and delivery

Hot reload
or restart

Trust store
and chain

Expiry probe
from outside

Renewal-failure alert
on the agent

Inventory and owner
per credential

Observation plane

Consumption plane

Distribution plane

Issuance plane

Public CA via ACME
or internal CA

Profile and policy
lifetime, SANs, key type

Renewal agent
certbot, cert-manager, SPIRE

Secret store
and delivery

Hot reload
or restart

Trust store
and chain

Expiry probe
from outside

Renewal-failure alert
on the agent

Inventory and owner
per credential

The issuance, distribution and consumption planes exist in every account. The observation plane is present in the designs and absent from the incidents. Boxes with a dashed border are the ones a typical enterprise estate does not have.
Diagram source

Issuance: the profile is a policy decision, not a default

What lifetime, which names, which key type, and which authority. Let's Encrypt's 90-day choice is documented as deliberate social engineering rather than cryptography: it is "short enough to strongly encourage operators to automate things (few people will want to manually renew certificates every quarter) but long enough to allow for manual renewal if necessary". Every internal CA makes the same choice implicitly, usually by copying a one-year default from a template nobody has revisited.

Documented at: ISRG, CCS '19, CA/Browser Forum

Distribution: the agent is a stateful thing that can get stuck

The renewal agent is usually treated as a cron job and behaves like a controller with state. cert-manager issue #6378 documents an agent that failed while its issuer was briefly down, then kept failing after the issuer recovered, widening its backoff from one hour to four to over thirteen, generating no new request and requiring a human to delete the failed object. An agent that cannot recover from a transient fault in its dependency is a second source of expiry, not a defence against the first.

Documented at: cert-manager #6378, cert-manager FAQ

Consumption: a new file on disk is not a new certificate in memory

The gap between the credential being replaced and the process using it is a real outage window. Cluster API's issue #10522 measures it: webhook failures for about 49 seconds, with the underlying cause that "it can take up to 60-90s for the new secret data to be propagated to the pod, while requests already try to get validated with the new secret", and it "usually happens every 60 days". Shorten the lifetime and you meet that window more often.

Documented at: cluster-api #10522

Observation: the plane that was missing in every incident

Two different alerts are needed and teams build at most one. An expiry probe checks the credential a client actually sees, from outside, and catches distribution and reload failures. A renewal-failure alert fires when the agent's own attempt fails, and is the only one with lead time. cert-manager exposes the second as a condition and a status.lastFailureTime field rather than a metric anyone alerts on by default. Bazel had neither, which is why a month of failures was invisible.

Documented at: Bazel postmortem, cert-manager FAQ

Internal machine identity is the same shape at a different scale

Uber's SPIFFE and SPIRE deployment covers "4,500 services running on hundreds of thousands of hosts across four clouds". At that size the binding constraint stops being expiry and becomes issuance capacity: each host caching "thousands of SVIDs" threatened memory, and an LRU cache let them register "around 2.5 times more workloads" while cutting "CPU usage by 40%". The lesson transfers downward. When you shorten lifetimes, you are multiplying load on the issuance plane, and that plane now has an availability target.

Documented at: Uber Engineering, 2023

The dependency nobody puts on the diagram

Short lifetimes convert a security property into an availability dependency, and RFC 8739 says so in plain text: "expiration replaces revocation", and "Reducing the caching properties... makes STAR clients increasingly dependent on the ACME server availability". Let's Encrypt halted all issuance for two and a half hours on 8 May 2026. At 398 days that is unnoticeable. At six days it is a third of your slack. The CA belongs on your dependency map with an SLO next to it.

Documented at: RFC 8739, Let's Encrypt status, May 2026

The renewal decision is moving from the client to the CA

The most consequential recent change is quiet and structural. RFC 9773, published in June 2025, defines ACME Renewal Information: the CA publishes a suggested renewal window per certificate and the client picks "a uniform random time within the suggested window". The stated motivation is load: fixed client-side renewal schedules mean "load clustering for the issuing CA due to its inability to schedule renewal requests", and they "create significant barriers against the issuing Certification Authority (CA) changing certificate lifetimes". Read that second clause carefully. It says the reason lifetimes have been hard to shorten is that renewal timing was hard-coded in millions of clients. ARI is the mechanism that unblocks the 47-day schedule.

It has a second use that matters more to an architect. The RFC notes that "a CA could suggest that clients renew prior to a mass-revocation event to mitigate the impact of the revocation", and in May 2026 Let's Encrypt did exactly that: after discovering its cross-signed intermediates were missing required Extended Key Usage fields, its final status update reads "Our ACME Renewal Information API is signalling affected certificates to renew now." A fleet on ACME with ARI enabled re-pulled itself. A fleet on annual manual renewal would have been a ticket queue.

03

The decisions that matter

Six forks with a published argument on both sides, including one where the industry consensus and the operational evidence pull in opposite directions.

Figure 4 · Choosing a lifetime, which is really choosing a slack

No

Yes

No

Yes

No

Yes

No

Yes

Does a named team own
this credential?

Stop. Assign an owner and
put it in an inventory first.

Is renewal fully automated,
end to end including reload?

Long lifetime, calendar reminder,
and treat it as tech debt with a date

Does a failed renewal
page someone?

Build the renewal-failure alert
before shortening anything

Can the workload survive
the reload window?

Fix hot reload first.
Shorter lifetimes hit it more often.

Short lifetime, renew at
1/3 to 1/2 of it, follow ARI

No

Yes

No

Yes

No

Yes

No

Yes

Does a named team own
this credential?

Stop. Assign an owner and
put it in an inventory first.

Is renewal fully automated,
end to end including reload?

Long lifetime, calendar reminder,
and treat it as tech debt with a date

Does a failed renewal
page someone?

Build the renewal-failure alert
before shortening anything

Can the workload survive
the reload window?

Fix hot reload first.
Shorter lifetimes hit it more often.

Short lifetime, renew at
1/3 to 1/2 of it, follow ARI

Every terminal node is an action. The left branch is the one most estates need and the one that gets deferred, because it is organisational rather than technical. Derived from the decision blocks below.
Diagram source

Decision 1: long lifetime with monitoring, or short lifetime with automation?

Chosen
  • Short with automation, by the entire public web PKI. Ballot SC-081v3 passed 25 to 0 among issuers.
  • Measured effect: renewal lapse fell from about 20% of trusted certificates before Let's Encrypt to 2.9% among its own renewals.
Rejected
  • Long lifetimes with human tracking, which is what most internal PKI still does.
  • Rejected because the interval is longer than the tenure of the person who set it, and because revocation does not work as a backstop.
Flips when
  • The consumer cannot be automated: an appliance, an embedded device, a partner's pinned client, or anything behind a change-approval board. Then the honest answer is a long lifetime, a named owner and a dated plan to remove it, not a short lifetime you will renew by hand.

Decision 2: who decides when renewal happens, the client or the authority?

Chosen
  • The authority, via ARI. RFC 9773 has the CA publish a suggested window and the client pick a uniform random time inside it.
  • Proved useful in anger in May 2026, when Let's Encrypt used ARI to pull forward renewals after a cross-sign defect.
Rejected
  • A fixed client-side schedule, which the RFC says causes "load clustering" and, more importantly, creates "significant barriers against the issuing Certification Authority (CA) changing certificate lifetimes".
Flips when
  • You run your own CA for internal workloads. Then you are both parties and the equivalent control is a server-side renewal policy your agents obey, plus deliberate jitter so ten thousand workloads do not renew at midnight together.

Decision 3: rotate on a schedule, or only when something forces it?

Chosen
  • On a schedule, and frequently, so the path is exercised. The industry position is that a rotation path used once a year is a rotation path that does not work.
Rejected
  • Rotate only on compromise. Rejected because revocation is unreliable: RFC 8739 states plainly that "expiration replaces revocation", and Let's Encrypt's own rationale for six-day certificates is that "certificate revocation doesn't work very well".
Flips when
  • The rotation itself is the risk. Azure's December 2025 outage was caused by an "inadvertent automated key rotation" against a component that expected manual, coordinated rotation. Automatic rotation of a credential whose consumer cannot handle rotation is worse than no rotation.

Decision 4: continuous renewal, or renewal windows?

Chosen
  • Continuous, in the protocol designs. ARI hands out randomised times specifically to spread load, and short lifetimes leave no room for a weekly window.
Rejected
  • Renewal windows, for a long time. cert-manager users asked for them in 2023; a pull request implementing a cron-expression window opened in November 2024 and was closed unmerged in May 2026 after eighteen months.
Flips when
  • Rotation causes user-visible disruption, which it does whenever reload is not hot. The demand did not go away: a redesign merged in March 2026 to "allow disabling or scheduling of certificate renewals" using cron and duration. If you need a window, your real problem is decision 5.

Decision 5: hot reload, or restart on renewal?

Chosen
  • Hot reload, everywhere it is available. The 2012 paper's prototype issued "a graceful restart" to Apache so it "loads the new certificates without disrupting any existing" connections.
Rejected
  • Rolling restart of the workload. It works, and it converts every renewal into a deployment, with a deployment's blast radius and change-control cost.
Flips when
  • The delivery path is slower than the trust update. Cluster API measured roughly 49 seconds of webhook failure because the API server trusted the new CA before the pod had the new secret. Hot reload does not help if the two sides update in the wrong order; the fix is to overlap validity, not to reload faster.

Decision 6: public CA, or internal CA, for internal traffic?

Chosen
  • Internal, at scale, with workload identity. Uber issues SVIDs to 4,500 services across four clouds through SPIRE rather than buying them.
  • The reason is not cost. It is that internal issuance can be made a local, high-availability service with lifetimes measured in hours.
Rejected
  • Public certificates for internal service-to-service traffic, which puts an external dependency and a public transparency log in the path of your service mesh.
Flips when
  • You have fewer than a few hundred workloads. Then running a CA is more risk than renting one, and the internal CA's own root expiry becomes the thing nobody is watching. Netflix built Lemur in 2015 precisely because organisations end up with both and can track neither.

Where the evidence pulls against the consensus

The consensus is unambiguous: shorter is better, automate everything. The operational evidence adds a caveat that the ballots do not. Every source that has measured rotation has found that rotation has a cost, and it is paid at rotation frequency. Cluster API pays about 49 seconds of webhook unavailability, currently every 60 days; at 47-day certificates that is eight times a year rather than six, and at six-day certificates it would be sixty. Azure paid a three-hour control-plane outage in two sovereign clouds because a rotation fired against a consumer that could not take it. The honest statement of the trade is that shortening lifetimes moves risk from the security column to the availability column, and the industry has decided that trade is worth making because revocation does not work. That is a defensible decision, and it is a decision, not a free improvement. If your workloads cannot take a rotation without a blip, the 47-day schedule is going to find out.

There is a second tension worth naming because it is invisible until it bites. Let's Encrypt ended expiration notification emails on 4 June 2025, and the reasoning is sound: after ten years "more and more of our subscribers have been able to put reliable automation into place", retaining millions of email addresses is a privacy liability, and the service cost "tens of thousands of dollars per year". But the email was the last human-legible signal for exactly the population that has not automated. Removing it is consistent with the forcing-function strategy and it removes a backstop from the estates least equipped to lose it. If your organisation was relying on that email, the mitigation is not to find another email service. It is to build the renewal-failure alert described in section 2.

DecisionChosenRejectedBecauseEvidence
Lifetime strategyShort plus automationLong plus human trackingLapse rate falls from about 20% to 2.9%CCS '19, SC-081v3
Renewal timing authorityThe CA, via ARIFixed client scheduleClient-side schedules block the CA from ever changing lifetimesRFC 9773
Renewal point1/3 to 1/2 of lifetime remainingA fixed number of days outA fixed offset does not scale down with the lifetimecert-manager, RFC 8739
Rotation triggerSchedule, frequentlyOnly on compromiseRevocation is unreliable, so expiry is the enforcement mechanismLet's Encrypt, RFC 8739
Renewal windowsContinuous with jitterCron-scheduled windowsShort lifetimes leave no room for a weekly windowcert-manager #7399
Applying a new credentialHot reload with overlapRolling restartRestart makes every renewal a deploymentcluster-api #10522
Internal service identityInternal CA plus workload identityPublic certificates internallyExternal dependency and public logging in the mesh pathUber
04

What broke in production

Ten published incidents in four classes. Only one class is the one people plan for, and it is not the one that produces the longest outages.

Figure 5 · The failure path, with the two silences that set the duration

Bazel usersManaged renewalDNSMaintainerBazel usersManaged renewalDNSMaintainercertificate still liststhat namerenewal fails. Nonotification. (silence1)2025-12-26 07:35certificate expiresno alert fires (silence2)15:08 investigationstarts2025-10-30 remove docs-stagingrecord2025-11-26 check every listeddomainname not reachableretries through December, allfailingbuilds fail worldwide09:00 reports on Slack and GitHub20:31 new certificate deployed, 13h total
Bazel usersManaged renewalDNSMaintainerBazel usersManaged renewalDNSMaintainercertificate still liststhat namerenewal fails. Nonotification. (silence1)2025-12-26 07:35certificate expiresno alert fires (silence2)15:08 investigationstarts2025-10-30 remove docs-stagingrecord2025-11-26 check every listeddomainname not reachableretries through December, allfailingbuilds fail worldwide09:00 reports on Slack and GitHub20:31 new certificate deployed, 13h total
Bazel, October 2025 to January 2026. The renewal machinery worked as designed and told nobody twice: once when renewal began failing, and once when the certificate expired. Reconstructed from the Bazel postmortem.
Diagram source

Class A: the renewal that failed silently

Postmortem

Bazel, *.bazel.build, December 2025

AssumptionThat a managed certificate from a cloud provider renews itself, and that a DNS cleanup is an unrelated change.
What happenedA subdomain was removed from DNS on 30 October while still listed on the certificate. Every listed domain must be reachable for renewal to succeed, so renewal began failing around 26 November and kept failing. "These failures never triggered any notifications."
Blast radiusAbout 13 hours. Bazel releases, the Bazel Central Registry and the source mirror were unavailable, so CI builds failed worldwide. The team had no alerting for certificate expiry.
FixA scheduled GitHub Actions workflow to detect impending expiry, a prober for general resource access, and a documented requirement that DNS changes account for certificates that reference the name.
Design ruleAny change that removes a name, a route or a permission must be checked against the certificates that assert it. The certificate is a distributed reference to your DNS zone and nothing enforces referential integrity.
Source

cert-manager: the agent that latched into failure

AssumptionThat a renewal agent recovers by itself once its issuer comes back.
What happenedThe issuer pod was briefly unavailable during a renewal. The request failed, and cert-manager kept failing with the same error after the issuer recovered, widening its backoff from one hour to four to more than thirteen, without generating a new request.
Blast radiusUnmeasured publicly, and that is the point: the certificate sits in a failed state until either a human deletes the failed request or it expires. Reported September 2023 and still open.
FixManual deletion of the CertificateRequest object. No automatic recovery is documented.
Design ruleAlert on the renewal agent's failure condition, not on days-to-expiry alone. Backoff means a stuck agent looks identical to a healthy idle one until the last day.

Class B: the rotation was the outage

Postmortem

Azure Government and Azure China, December 2025

AssumptionThat automatic key rotation is safe to leave enabled, and that a consumer expecting manual rotation would be configured accordingly.
What happenedCosmos DB accounts backing Azure Resource Manager were "unintentionally configured with the option to automatically rotate the keys enabled" while ARM used the keys in manual mode. The rotation fired and authorization failed.
Blast radiusService management operations failed across all Azure Government regions from 11:04 to 14:13 EST on 8 December 2025, with an equivalent failure in Azure China about three hours earlier. The keys for both clouds "were created the same day in February 2025, starting completely independent timers", and Microsoft states the team "did not have a sufficient understanding of the inadvertent key rotation to be able to prevent impact in the second sovereign cloud".
FixAn audit for misconfigured manual keys across all clouds, automated rotations paused pending usage signals, a migration to auto-rotated keys, and change-safety controls on key regeneration.
Design ruleRotation mode is a contract between the issuer and every consumer of a credential; if any consumer pins, caches or coordinates manually, automatic rotation upstream is a scheduled outage. And credentials provisioned in the same operation share a timer even when the systems are isolated, so record the provisioning date, not only the expiry.
Source

Cluster API: the reload window nobody budgets for

AssumptionThat rotating a webhook's serving certificate is transparent because cert-manager handles it.
What happenedThe API server begins trusting the new CA immediately, while the pod receives the new secret through a volume mount that "can take up to 60-90s". Requests validated in that gap fail. Observed failures ran about 49 seconds, and the issue notes it "usually happens every 60 days".
Blast radiusAdmission webhook failures, which in a Kubernetes cluster means writes are rejected. Reported May 2024 and open.
FixNone merged. The workable approaches are overlapping validity and having the consumer watch the secret rather than the filesystem.
Design ruleOrder matters more than speed. Trust the new issuer before you serve the new credential, and keep the old one valid until the new one is confirmed in use everywhere.

Class C: somebody else changed the terms

Postmortem

DigiCert: 83,267 certificates, 24 hours' notice, July 2024

AssumptionThat a certificate you hold stays valid until its expiry date, and that rotation is something you schedule.
What happenedA 2019 refactor left one validation path that did not add the required underscore prefix to a CNAME random value. A third party reported it on 3 July 2024. Because the defect touched domain validation, the CA/Browser Forum's 24-hour revocation rule applied, and DigiCert notified 6,807 subscribers that certificates covering 172,047 domains would be revoked.
Blast radiusAll 83,267 certificates revoked by 3 August 2024, five days rather than 24 hours. DigiCert stated that customers "operating critical infrastructure, vital telecommunications networks, cloud services, and healthcare industries are not in a position to be revoked without critical service interruptions".
FixCompliance in every architecture review, removal of product teams' ability to self-identify compliance-relevant changes, test coverage for random value structure, elimination of infrequently used validation paths, and open-sourcing the domain validation system.
Design ruleYour rotation lead time is not a choice you own. Assume 24 hours, because that is the rule your CA is bound by, and rehearse a full estate reissue before someone else schedules it for you.
Postmortem

A subscriber went to court to stop a revocation

AssumptionThat certificate lifecycle automation is an operations concern with an operations-sized consequence.
What happenedOn 30 July 2024 a DigiCert customer obtained a Temporary Restraining Order prohibiting the revocation of its certificates, because it could not replace them inside the mandated window. DigiCert confirmed in the public incident report: "We did receive a TRO in connection with this revocation."
Blast radiusThe revocation of a security defect was delayed by litigation. The CA's chief compliance officer resigned in the aftermath.
FixNone available to the subscriber except the automation it did not have.
Design ruleThis is the sentence for the risk register: an organisation whose certificate rotation is manual has a business continuity exposure whose worst case is a court filing. Rotation capability is a control, not a chore, and it belongs in continuity testing beside failover.
Postmortem

Let's Encrypt halts all issuance, May 2026

AssumptionThat the certificate authority is always there, which is safe at 398 days and an assumption at six.
What happenedAt 18:37 UTC on 8 May 2026: "We have been made aware of a potential incident and are shutting down all issuance." Issuance resumed at 21:03 after switching back to the previous root. The defect was that cross-certified subordinate CAs were "missing Extended Key Usage (EKU) fields which are now required".
Blast radiusAbout two and a half hours with no issuance at all, then a five-day tail while the cross-signs were revoked and reissued. End-entity certificates were not revoked.
FixRevoke and reissue the cross-signs, and use the renewal API to pull affected fleets forward: "Our ACME Renewal Information API is signalling affected certificates to renew now."
Design rulePut the CA on the dependency map with a stated availability budget, and size the slack so that a plausible CA outage is a fraction of it rather than a multiple. Two and a half hours against two days of slack is fine; against four hours it is not.
Eng blog

The warning email that was switched off, June 2025

AssumptionThat the certificate authority will keep telling you when something is about to expire.
What happenedLet's Encrypt ended expiration notification emails on 4 June 2025, having announced it in January. The reasoning is sound on its own terms: after ten years "more and more of our subscribers have been able to put reliable automation into place", retaining millions of email addresses is a privacy liability, and the service cost "tens of thousands of dollars per year".
Blast radiusNot an incident, and included because it removes a control. The population still relying on that email is precisely the population without the automation that made it redundant.
FixLet's Encrypt suggests a third-party monitoring service. The durable answer is the renewal-failure alert in section 2, which fires earlier and does not depend on anyone's mailing list.
Design ruleAudit which of your alerts are actually somebody else's goodwill. A notification you do not own, do not pay for and cannot test is not a control, and it can be withdrawn with five months' notice.

Class D: nobody owned it

Eng blog

Ericsson, O2 and SoftBank, December 2018

AssumptionThat a certificate embedded in shipped software by a vendor is the vendor's problem, and that the operator running it has visibility into it.
What happenedAn expired certificate in two specific software versions of Ericsson's packet core caused simultaneous failures in mobile networks on two continents. Ericsson's chief executive said the software "is being decommissioned".
Blast radiusRoughly 32 million UK subscribers on O2 and its virtual operators lost data service, with a comparable nationwide failure at SoftBank in Japan, on the same day.
FixReported as decommissioning the affected software versions. No first-party engineering account survives online.
Design ruleCredentials inside vendor-supplied software are the ones with no owner in your organisation and no inventory entry. Ask every vendor for the expiry dates embedded in what they ship, and treat a refusal as a finding.
SourceTechCrunch, 7 December 2018, reported rather than first-party
Eng blog

The problem Netflix named in 2015 and nobody has solved

AssumptionThat certificates have owners because someone requested them.
What happenedNetflix released Lemur because certificate procurement was "handled by humans" across internal and external authorities with no common record, and because "Certificates have expiration dates: if they are allowed to expire without replacing communication can be interrupted, impacting a system's availability". Eleven years on, only 32% of organisations report using a dedicated lifecycle management tool.
Blast radiusIndustry-wide and continuous: an average of nine certificate-related incidents per organisation per year, at 2.6 hours to identify and 2.7 hours to remediate, involving eight staff.
FixAn inventory with an owner per credential. This is the least technical item in the guide and the one most estates still lack.
Design ruleA credential with no named owner has no renewal process, regardless of what tooling exists. Build the inventory before the automation, because automation you cannot attribute is automation you cannot alert on.
05

Numbers you can plan against

Everything quantitative in this guide, with the organisation, the context it was measured in, and the date. Read the note under the table before any of it reaches a capacity or risk model.

Figure 6 · What the schedule does to your slack if you change nothing else

0 40 80 120 Slack (days) 398-day 200-day 100-day 47-day until 2026 Mar 2026 Mar 2027 Mar 2029 133 67 33 16 30 30 30 30 Renew at 1/3 remaining Fixed 30 days out
Derived, not reported: the blue bars are one third of each lifetime, which is the cert-manager default; the gold bars are a fixed 30-day-out renewal, which is the common enterprise pattern. At 47 days the fixed offset consumes almost two thirds of the certificate's life and the two strategies invert. Lifetimes from ballot SC-081v3.
MetricValueAtContextAs ofSource
Time to identify a certificate-related outage2.6 hSurveyBefore remediation, which adds another 2.7 h2024-04Keyfactor
Staff involved in remediating one outage8SurveyAverage across respondents2024-04Keyfactor
Certificate-related incidents per organisation per year9SurveyIncludes outages, failed audits and breaches2024-04Keyfactor
Organisations using a dedicated lifecycle management tool32%SurveyThe rest track certificates by other means2024-04Keyfactor
Silent renewal failure before expiry~30 daysBazelRenewal failing from about 26 Nov, expiry 26 Dec2025-12Bazel
Outage duration once the certificate expired13 hBazel07:35 expiry to 20:31 replacement, users reporting from 09:002025-12Bazel
Certificates revoked on a forced rotation83,267DigiCert6,807 subscribers, 172,047 unique domains2024-08Bugzilla 1910322
Mandated revocation window for a validation defect24 hCA/B ForumActual completion took five days, with a court order in between2024-08Bugzilla 1910322
Complete issuance outage at a major CA2 h 26 mLet's Encrypt18:37 to 21:03 UTC, all issuance halted2026-05Status page
Control-plane outage from an unwanted key rotation3 h 9 mAzure Government11:04 to 14:13 EST, repeated in Azure China three hours earlier2025-12Azure PIR
Webhook unavailability during certificate rotation~49 sCluster APISecret propagation to the pod takes "up to 60-90s"; recurs every 60 days2024-05cluster-api #10522
Renewal lapse rate before automation~20%Web PKIOf trusted certificates, pre-Let's Encrypt2019-11CCS '19
Let's Encrypt renewals occurring after expiry2.9%Let's Encrypt2.2% of Top Million LE sites serve expired certificates, against 3.9% of all HTTPS sites2019-11CCS '19
Certificates issued in the first three years of automated issuance538 MLet's EncryptCovering 223 million domains2019-11CCS '19
Maximum public certificate validity, current tier200 daysCA/B ForumFrom 15 March 2026; 100 days from 2027, 47 days from 20292026-03SC-081v3
Domain validation data reuse, final tier10 daysCA/B ForumDown from 398 days; revalidation, not just reissuance2029-03SC-081v3
Short-lived certificate profile6 daysLet's EncryptRenew "every two to three days", ARI checked at least daily2025-02Let's Encrypt
Required publication point for the next credentialhalf-lifeIETF"halfway through the lifetime of the currently active certificate at the latest"2020-03RFC 8739
Default renewal point2/3 of durationcert-managerLeaving one third of the lifetime as slack; retries back off 1 to 32 hourscurrentcert-manager FAQ
Web PKI certificates issued through ACME>50%Chrome Root ProgramAbout 95% come from a CA owner that offers ACME in some formcurrentChrome
Internal workload identity estate4,500 servicesUber"hundreds of thousands of hosts across four clouds"; hosts cache thousands of SVIDs each2023-11Uber
Issuance efficiency gain from caching workload identities2.5× / 40%Uber2.5 times more workloads registered per host group, 40% less CPU on the SPIRE server2023-11Uber
Subscribers affected by one expired vendor certificate~32 MO2 UKPlus a nationwide SoftBank failure the same day; reported figure2018-12TechCrunch
Read these carefully

Measured: the incident timelines and counts from Bazel, DigiCert, Azure, Let's Encrypt and Cluster API, and the Let's Encrypt research figures, which were computed from internet-wide scans. Vendor-collected: every Keyfactor figure. They are the best public numbers on the industry-wide rate of this failure and they come from a company that sells certificate lifecycle management, so treat the direction as reliable and the magnitude as soft. There is no independent equivalent. Derived: the slack figures in Figure 6 are arithmetic on the ballot schedule, not measurements. Reported rather than documented: the Ericsson subscriber count, which comes from journalism because no first-party account remains online.

Unknown, and this is where your risk is: nobody has published the expiry-incident rate of an internal PKI before and after automation, so there is no public figure for what the investment buys. And no organisation has published an operational account of running a large estate at 100-day or 47-day certificates, because those tiers begin in 2027 and 2029. The 200-day tier has five months of history. Everything anyone tells you about operating at 47 days is currently a prediction.

How to work out your own slack this afternoon

The calculation needs three numbers per credential and none of them require new tooling. Take the lifetime L from the certificate. Take the renewal point from your agent's configuration, which gives you the window W; if it is expressed as a fixed number of days, write down that number, and if it is a fraction, multiply. Take the retry interval R, remembering that agents back off, so use the cap: cert-manager's is 32 hours, and a managed cloud certificate that renews once has an effective R equal to W, which is a slack of one. Then W divided by R is the number of consecutive failures you survive.

Do it for your ten most important credentials and the distribution will be bimodal. The ones under a modern agent will have a slack in the tens. The ones behind a managed service, a vendor appliance or a wiki page will have a slack of one, and those are the ones that will fail, because a slack of one means any single transient fault in a dependency of the renewal path is an outage scheduled for a date you can already calculate. Put that date on the dashboard next to the credential. It is the most useful thing an architect can add to this problem in an afternoon, and it costs nothing.

06

The evidence wall

Every source behind this page, graded, with what it shows and the one thing to carry out of it. Filter by kind. The full ledger, with the quote supporting each individual claim, ships beside this file as sources.md.

Postmortem Bazel2026-01-16

Postmortem for *.bazel.build SSL certificate expiry

The best recent subscriber-side account in public, and unusually honest about the human timeline. A DNS cleanup two months earlier broke a managed renewal that then failed for a month without notifying anyone, and the response took thirteen hours because nobody was watching and the first responder was in the wrong timezone.

Carry forwardRemoving a DNS name is a change to every certificate that lists it. Nothing enforces that link.
blog.bazel.build/2026/01/16/ssl-cert-expiry.html
Postmortem DigiCert2024-07

Mozilla Bugzilla 1910322: random value in CNAME without underscore prefix

The single richest document in this guide. Root-store policy obliges certificate authorities to file public incident reports, and this one runs to the full argument: a 2019 refactor, a five-year latent defect, 83,267 certificates, a 24-hour revocation mandate, a court order from a subscriber who could not comply, and the compliance officer's resignation.

Carry forwardRead CA incident reports. It is a corpus of detailed, adversarially reviewed postmortems that almost no architect knows exists.
bugzilla.mozilla.org/show_bug.cgi?id=1910322
Postmortem Microsoft Azure2025-12

Post Incident Review, tracking ID ML7_-DWG

A first-party account of automated rotation causing an outage rather than preventing one, in two sovereign clouds three hours apart, because keys created on the same day in February 2025 started independent timers that then fired independently. Includes the admission that the first incident did not produce enough understanding to stop the second.

Carry forwardCredentials provisioned together share a timer. Record the provisioning date, not only the expiry.
azure.status.microsoft/status/history/?trackingId=ML7_-DWG
Postmortem Let's Encrypt2026-05

Incident of 8 May 2026: cross-sign missing Extended Key Usage

Two and a half hours with no issuance at all from the authority that issues most of the web's certificates, then a five-day tail of revoking and reissuing cross-signs. The resolution update is the interesting part: the renewal API was used to pull affected fleets forward rather than emailing anyone.

Carry forwardYour CA is a runtime dependency once lifetimes are short. Give it a line on the dependency map and a number.
letsencrypt.status.io · incident 2026-05-08
Decision record CA/Browser Forum2025-04-11

Ballot SC-081v3: schedule of reducing validity and data reuse periods

The decision that sets the deadline everyone is now working to. Validity falls from 398 days to 47 across three steps between March 2026 and March 2029, and domain validation data reuse falls from 398 days to 10. Proposed by Apple, endorsed by Sectigo, Google and Mozilla, and passed with no votes against.

Carry forwardThe 10-day validation reuse tier is the sleeper. It forces revalidation, not just reissuance, which breaks air-gapped and manual approval flows.
cabforum.org · ballot SC-081v3
Decision record Google Chromecurrent

Chrome Root Program: Moving Forward, Together

The policy document that states the strategy out loud. Reducing lifetime "encourages automation and the adoption of practices that will drive the ecosystem away from baroque, time-consuming, and error-prone issuance processes", and Chrome intends to require that root store applicants offer ACME.

Carry forwardIf your CA does not offer ACME, its position in the root stores is a strategic risk to you, not just to it.
chromium.googlesource.com · moving-forward-together
Decision record IETF / ISRG2025-06

RFC 9773: ACME Renewal Information (ARI)

Moves the renewal decision from the client to the authority, with the client choosing a uniform random time inside a suggested window. The introduction is candid that hard-coded client renewal schedules were the thing blocking the industry from shortening lifetimes at all.

Carry forwardEnable ARI in your client. It is how a CA tells your fleet to renew early during an incident, and it is the only channel that scales.
rfc-editor.org/rfc/rfc9773.html
Decision record IETF2020-03

RFC 8739: Short-Term, Automatically Renewed (STAR) certificates

The design that states the trade in one line, "expiration replaces revocation", and then states its cost in the next: short lifetimes make clients "increasingly dependent on the ACME server availability". Also the source of the formal slack rule, that the next certificate must be available by half-life at the latest.

Carry forwardPublish the replacement by half-life. That single rule turns a renewal failure into a warning rather than an outage.
datatracker.ietf.org/doc/html/rfc8739
Paper Stanford & CMU2012

Towards Short-Lived Certificates (W2SP '12)

Where the whole trajectory starts, opening with "The Online Certificate Status Protocol (OCSP) is as good as dead." Proposes four-day certificates fetched daily, and states the slack principle explicitly: a failed fetch is harmless because the previous certificate is still valid, giving everyone time to fix the problem.

Carry forwardThe design goal of a short lifetime is not a short lifetime. It is a renewal you can afford to have fail.
ieee-security.org/TC/W2SP/2012/papers/w2sp12-final9.pdf
Paper ISRG2019-11

Let's Encrypt: An Automated Certificate Authority to Encrypt the Entire Web (CCS '19)

The measurement that settles the argument. Renewal lapse affected about 20% of trusted certificates before Let's Encrypt; 2.9% of its own renewals happen after expiry. It also documents that the 90-day lifetime was chosen as a behavioural forcing function rather than for a cryptographic reason.

Carry forwardCite this when someone argues that shorter lifetimes will cause more outages. The measured effect is the opposite, conditional on automation.
abetterinternet.org/documents/letsencryptCCS2019.pdf
Eng blog Let's Encrypt2025-01-22

Ending Support for Expiration Notification Emails

Josh Aas explaining why the last human-legible warning was switched off on 4 June 2025: automation is now widespread, the email addresses are a privacy liability, and the service cost tens of thousands of dollars a year. Consistent with the strategy, and it removes a backstop from exactly the estates that still need one.

Carry forwardIf a person in your organisation was the alerting mechanism, you no longer have one. Build the renewal-failure alert.
letsencrypt.org/2025/01/22/ending-expiration-emails
Eng blog Let's Encrypt2025-02-20

We Issued Our First Six Day Cert

The operational guidance that comes with a six-day lifetime, and the clearest statement of renewal cadence anywhere: renew "every two to three days", check the renewal API "at least once per day". That is a slack of two to three attempts, deliberately chosen.

Carry forwardRenewal cadence scales with lifetime. A fixed 30-day-out renewal is meaningless below about 90 days.
letsencrypt.org/2025/02/20/first-short-lived-cert-issued
Eng blog Uber2023-11-09

Our Journey Adopting SPIFFE/SPIRE at Scale

What internal machine identity looks like at 4,500 services across four clouds and hundreds of thousands of hosts. The problems are not cryptographic: caching thousands of identities per host, signing load on the identity server, and false positives in agent health checks under load.

Carry forwardShortening lifetimes multiplies issuance load. Size the issuance plane as a production service with its own SLO.
uber.com/en-IE/blog/our-journey-adopting-spiffe-spire/
Eng blog Netflix2015-09-21

Introducing Lemur

Worth reading now precisely because it is eleven years old and describes today's problem: procurement handled by humans, internal and external authorities with no common record, and expiry as an availability risk. The tooling changed and the ownership gap did not.

Carry forwardThe inventory with a named owner per credential is the prerequisite, not the follow-up.
netflixtechblog.com/introducing-lemur-ceae8830f621
Eng blog Keyfactor2024-04-08

Key takeaways from the 2024 PKI and Digital Trust Report

The only public source of industry-wide rates for this failure, and a vendor survey, so weigh it accordingly. The figure that matters is not the incident count: it is 2.6 hours to identify an outage against 2.7 hours to remediate it, which says detection is half the problem.

Carry forwardHalf your mean time to recovery in this class is spent working out that a credential expired. That half is the cheapest to remove.
keyfactor.com/blog/key-takeaways-from-the-2024-pki-digital-trust-report/
Eng blog TechCrunch2018-12-07

Here's what caused yesterday's O2 and SoftBank outages

Reported rather than first-party, and included because it is the largest published blast radius in this class and because the absence of an Ericsson engineering account is itself the finding. Roughly 32 million UK subscribers and a nationwide Japanese outage, from certificates inside shipped vendor software.

Carry forwardAsk vendors for the expiry dates embedded in what they ship you. Nobody in your organisation owns those and nobody can see them.
techcrunch.com/2018/12/07/heres-what-caused-yesterdays-o2-and-softbank-outages
Source cert-managercurrent

cert-manager FAQ: renewal timing and terminal failure

The defaults most Kubernetes estates are running without having read them. Renewal at two thirds of the actual duration, terminal failure visible only as a condition and a status.lastFailureTime field, and retries backing off from one hour to a cap of thirty-two.

Carry forwardAlert on lastFailureTime being set. It is the earliest signal you get and nothing surfaces it by default.
cert-manager.io/docs/faq/
Source cert-manager2023-09-28

Issue #6378: renewal fails during issuer downtime and keeps failing

A renewal agent that cannot recover from a transient fault in its own dependency. The backoff widens from one hour to four to more than thirteen, no new request is generated, and a human has to delete the failed object. Open since September 2023.

Carry forwardA stuck agent is indistinguishable from a healthy idle one on a days-to-expiry dashboard. Watch the agent, not only the clock.
github.com/cert-manager/cert-manager/issues/6378
Source cert-manager2024-11 to 2026-05

PR #7399: add a renewal window, closed unmerged after eighteen months

Operators asked for cron-scheduled renewal windows so rotations happen outside business hours. The pull request sat from November 2024 and was closed in May 2026 with "This feature is now implemented in #8258". The replacement merged in March 2026. The gap is the argument: continuous renewal is the protocol direction and change windows are the operational reality.

Carry forwardIf you need a renewal window, the underlying problem is that rotation is user-visible. Fix the reload path instead.
github.com/cert-manager/cert-manager/pull/7399
Source cert-manager2026-03-23

PR #8258: certificate renewal policy and windows, merged

The redesign that did land, "to allow disabling or scheduling of certificate renewals" using a cron start and a duration. Worth reading next to #7399 as a record of how long a contested operational feature takes to reach a release when it cuts against the project's direction.

Carry forwardThe capability now exists upstream. Use it to schedule, not to defer; a window that skips a cycle silently spends your slack.
github.com/cert-manager/cert-manager/pull/8258
Source Kubernetes SIG Cluster Lifecycle2024-05

cluster-api issue #10522: rotation causes webhook downtime

Measures the cost of the thing everyone assumes is free. The API server trusts the new CA immediately while the pod waits up to 60 to 90 seconds for the secret to propagate, producing about 49 seconds of admission failures, roughly every 60 days.

Carry forwardTrust the new issuer before you serve the new credential, and keep the previous one valid until the new one is confirmed in use.
github.com/kubernetes-sigs/cluster-api/issues/10522
Talk IETF 1212024-11-06

ACME Renewal Information and profiles, working group slides

Aaron Gable of Let's Encrypt presenting draft-ietf-acme-ari-06 to the ACME working group, alongside the certificate profiles proposal that later let Let's Encrypt offer a six-day profile at all. Included partly because it is one of very few citable talks on this topic.

Carry forwardThe design argument for this subject happens in IETF drafts and CA/Browser Forum ballots, not on conference stages. Read the ballots.
datatracker.ietf.org/meeting/121/materials/slides-121-acme-ari-and-profiles-00
Where the record runs out

Five gaps, named because they are where your risk sits. First, no first-party Ericsson account survives, so the largest published blast radius in this class is known only through journalism. Second, the talk layer is genuinely thin: there is no conference corpus to match the postmortems, because the argument lives in ballots and bug trackers. Third, nobody has published a measured before-and-after for internal PKI automation, so the return on that investment has no public number. Fourth, the industry-rate figures are vendor-collected and there is no independent equivalent. Fifth, and most important for anyone planning: the 100-day and 47-day tiers have no operational history at all, because they begin in 2027 and 2029. The 200-day tier is five months old. Anyone describing how a large estate behaves at 47 days is predicting, including this guide.

07

Build a miniature, then productionise it

Six rungs. The first three run on a laptop in an evening. The crossing into production shape is rung four, where you stop building the renewal and start building the thing that watches it.

Issue, expire, and watch it break

Stand up a local CA and issue a certificate with a two-minute lifetime to a service behind a client that validates it. Watch the client fail at the second it expires.

Done when: you can produce the exact client-side error your production stack would emit, and you know which log it lands in.  Teaches: what your on-call actually sees, which in most stacks is a generic TLS handshake failure with no mention of dates.

Automate the renewal, then break the automation

Add an agent that renews at half-life. Then take the CA offline for longer than one retry interval and watch what the agent does. Then take it offline for longer than the whole window.

Done when: you can state your slack as a number of attempts and demonstrate the two outcomes either side of it.  Teaches: that renewal failure and expiry are separated by a window you chose, usually by accident.

Make the failure silent, then make it loud

Confirm that nothing in your setup reports the failed renewal. Then add two alerts: an external probe on the served certificate's remaining validity, and an alert on the agent's own failure condition. Break renewal again and check which fires first.

Done when: the agent alert fires days before the probe alert would.  Teaches: why the Bazel incident ran for a month invisibly, and why an expiry dashboard alone is a lagging indicator.

Rotate under load and measure the gap

Put steady traffic through the service and force a rotation. Measure the error window between the new credential landing and the process serving it. Then fix it: overlap validity, reload without restart, and update the trust store before the serving certificate.

Done when: a rotation under load produces zero failed requests, and you have the before number written down.  Teaches: the cost Cluster API measures at about 49 seconds, and that ordering matters more than speed.

Simulate the forced rotation

Pick your production estate and answer, on paper, how you would replace every certificate from one authority within 24 hours. Then do it for real in a non-production environment. Count the ones that need a change ticket, a vendor, or a person on holiday.

Done when: you have a number for full-estate reissue time and a named list of the credentials that cannot make 24 hours.  Teaches: the exposure that put a DigiCert customer in front of a judge in 2024.

Publish the slack, and put a date on the exceptions

Compute lifetime, renewal window and retry interval for the top credentials in your real estate, publish the slack next to each, and for every credential with a slack of one, record the calendar date on which a single transient fault becomes an outage. Assign an owner to each.

Done when: the list exists, has owners, and has been read by someone who can fund the work.  Teaches: that this failure class is an inventory and ownership problem wearing a cryptography costume.

08

Keep hunting

The queries that found this material. The highest-yield move on this topic is not a search engine at all: certificate authorities are obliged to file public incident reports, and that corpus is sitting in a bug tracker.

The corpus nobody reads

  • site:bugzilla.mozilla.org "incident report" CA revocation
  • bugzilla.mozilla.org product:"CA Program" "delayed revocation"
  • "certificate problem report" "24 hours" subscriber impact
  • site:cabforum.org ballot validity OR revocation OR automation

Subscriber-side incidents, in their own words

  • "expired certificate" postmortem "never triggered any notifications"
  • intitle:postmortem "certificate" expiry blog -tutorial -guide
  • "post incident review" "key rotation" root cause site:azure.status.microsoft
  • status page history "we have shut down all issuance"

The design argument

  • datatracker.ietf.org acme "renewal information" OR "short-term" draft
  • "expiration replaces revocation" OR "halfway through the lifetime"
  • repo:cert-manager/cert-manager is:pr is:closed is:unmerged renew
  • "renewBefore" OR "renewal window" issue rotation downtime webhook

Scale and internal PKI

  • intitle:"our journey" SPIFFE OR SPIRE OR "workload identity" at scale
  • engineering blog "internal PKI" "we built" certificate rotation numbers
  • "short-lived credentials" production "we moved" mTLS -vendor
  • "certificate lifecycle" survey outage hours identify remediate
09

References

  1. Xudong Yang. Postmortem for *.bazel.build SSL certificate expiry Bazel blog, 16 January 2026. Checked 2026-08-29.
  2. Bug 1910322: DigiCert, random value in CNAME without underscore prefix Mozilla Bugzilla, July to August 2024. Checked 2026-08-29.
  3. Post Incident Review, tracking ID ML7_-DWG Microsoft Azure status history, December 2025. Checked 2026-08-29.
  4. Incident of 8 May 2026, issuance halted Let's Encrypt status, 8 to 13 May 2026. Checked 2026-08-29.
  5. Ballot SC081v3: Introduce schedule of reducing validity and data reuse periods CA/Browser Forum, 11 April 2025. Checked 2026-08-29.
  6. Chrome Root Program: Moving Forward, Together Chromium website source, current at retrieval. Checked 2026-08-29.
  7. A. Gable. RFC 9773: ACME Renewal Information (ARI) Extension Internet Security Research Group, June 2025. Checked 2026-08-29.
  8. Y. Sheffer and others. RFC 8739: Support for Short-Term, Automatically Renewed (STAR) Certificates in ACME IETF, March 2020. Checked 2026-08-29.
  9. Topalovic, Saeta, Huang, Jackson, Boneh. Towards Short-Lived Certificates Web 2.0 Security and Privacy (W2SP), 2012. Checked 2026-08-29.
  10. Aas and others. Let's Encrypt: An Automated Certificate Authority to Encrypt the Entire Web ACM CCS '19, London, November 2019. Checked 2026-08-29.
  11. Josh Aas. Ending Support for Expiration Notification Emails Let's Encrypt, 22 January 2025. Checked 2026-08-29.
  12. Josh Aas. We Issued Our First Six Day Cert Let's Encrypt, 20 February 2025. Checked 2026-08-29.
  13. Haque, Borole, Raja, Train, Nusbaum, Turner, Moore. Our Journey Adopting SPIFFE/SPIRE at Scale Uber Engineering, 9 November 2023. Checked 2026-08-29.
  14. Glisson, Chan, Hagen. Introducing Lemur Netflix TechBlog, 21 September 2015. Checked 2026-08-29.
  15. Key takeaways from the 2024 PKI and Digital Trust Report Keyfactor, 8 April 2024. Vendor survey. Checked 2026-08-29.
  16. Here's what caused yesterday's O2 and SoftBank outages TechCrunch, 7 December 2018. Reported, not first-party. Checked 2026-08-29.
  17. cert-manager FAQ: renewal timing and terminal failure cert-manager documentation, current at retrieval. Checked 2026-08-29.
  18. cert-manager issue #6378: renewal fails during issuer downtime, continues to fail after recovery Opened 28 September 2023. Checked 2026-08-29.
  19. cert-manager PR #7399: add renew window to restrict when certificate renewal can happen Opened 5 November 2024, closed unmerged 29 May 2026. Checked 2026-08-29.
  20. cert-manager PR #8258: certificate renewal policy and windows Opened 16 November 2025, merged 23 March 2026. Checked 2026-08-29.
  21. cluster-api issue #10522: cert-manager certificate rotation may lead to webhook downtime of up to 90s Kubernetes SIG Cluster Lifecycle, May 2024. Checked 2026-08-29.
  22. Aaron Gable. ACME Renewal Information, draft-ietf-acme-ari-06, and ACME Profiles IETF 121 ACME working group, 6 November 2024. Checked 2026-08-29.