Every source behind this page, graded, with what it shows and the one thing
to carry out of it. Filter by kind. The full ledger, with the quote supporting each
individual claim, ships beside this file as sources.md.
Postmortem
Bazel2026-01-16
Postmortem for *.bazel.build SSL certificate expiry
The best recent subscriber-side account in public, and unusually honest about the human
timeline. A DNS cleanup two months earlier broke a managed renewal that then failed for a
month without notifying anyone, and the response took thirteen hours because nobody was
watching and the first responder was in the wrong timezone.
Carry forwardRemoving a DNS name is a change to every certificate that lists it. Nothing enforces that link.
blog.bazel.build/2026/01/16/ssl-cert-expiry.html
Postmortem
DigiCert2024-07
Mozilla Bugzilla 1910322: random value in CNAME without underscore prefix
The single richest document in this guide. Root-store policy obliges certificate
authorities to file public incident reports, and this one runs to the full argument: a
2019 refactor, a five-year latent defect, 83,267 certificates, a 24-hour revocation
mandate, a court order from a subscriber who could not comply, and the compliance
officer's resignation.
Carry forwardRead CA incident reports. It is a corpus of detailed, adversarially reviewed postmortems that almost no architect knows exists.
bugzilla.mozilla.org/show_bug.cgi?id=1910322
Postmortem
Microsoft Azure2025-12
Post Incident Review, tracking ID ML7_-DWG
A first-party account of automated rotation causing an outage rather than preventing
one, in two sovereign clouds three hours apart, because keys created on the same day in
February 2025 started independent timers that then fired independently. Includes the
admission that the first incident did not produce enough understanding to stop the
second.
Carry forwardCredentials provisioned together share a timer. Record the provisioning date, not only the expiry.
azure.status.microsoft/status/history/?trackingId=ML7_-DWG
Postmortem
Let's Encrypt2026-05
Incident of 8 May 2026: cross-sign missing Extended Key Usage
Two and a half hours with no issuance at all from the authority that issues most of the
web's certificates, then a five-day tail of revoking and reissuing cross-signs. The
resolution update is the interesting part: the renewal API was used to pull affected
fleets forward rather than emailing anyone.
Carry forwardYour CA is a runtime dependency once lifetimes are short. Give it a line on the dependency map and a number.
letsencrypt.status.io · incident 2026-05-08
Decision record
CA/Browser Forum2025-04-11
Ballot SC-081v3: schedule of reducing validity and data reuse periods
The decision that sets the deadline everyone is now working to. Validity falls from 398
days to 47 across three steps between March 2026 and March 2029, and domain validation
data reuse falls from 398 days to 10. Proposed by Apple, endorsed by Sectigo, Google and
Mozilla, and passed with no votes against.
Carry forwardThe 10-day validation reuse tier is the sleeper. It forces revalidation, not just reissuance, which breaks air-gapped and manual approval flows.
cabforum.org · ballot SC-081v3
Decision record
Google Chromecurrent
Chrome Root Program: Moving Forward, Together
The policy document that states the strategy out loud. Reducing lifetime "encourages
automation and the adoption of practices that will drive the ecosystem away from
baroque, time-consuming, and error-prone issuance processes", and Chrome intends to
require that root store applicants offer ACME.
Carry forwardIf your CA does not offer ACME, its position in the root stores is a strategic risk to you, not just to it.
chromium.googlesource.com · moving-forward-together
Decision record
IETF / ISRG2025-06
RFC 9773: ACME Renewal Information (ARI)
Moves the renewal decision from the client to the authority, with the client choosing a
uniform random time inside a suggested window. The introduction is candid that
hard-coded client renewal schedules were the thing blocking the industry from shortening
lifetimes at all.
Carry forwardEnable ARI in your client. It is how a CA tells your fleet to renew early during an incident, and it is the only channel that scales.
rfc-editor.org/rfc/rfc9773.html
Decision record
IETF2020-03
RFC 8739: Short-Term, Automatically Renewed (STAR) certificates
The design that states the trade in one line, "expiration replaces revocation", and
then states its cost in the next: short lifetimes make clients "increasingly dependent
on the ACME server availability". Also the source of the formal slack rule, that the
next certificate must be available by half-life at the latest.
Carry forwardPublish the replacement by half-life. That single rule turns a renewal failure into a warning rather than an outage.
datatracker.ietf.org/doc/html/rfc8739
Paper
Stanford & CMU2012
Towards Short-Lived Certificates (W2SP '12)
Where the whole trajectory starts, opening with "The Online Certificate Status Protocol
(OCSP) is as good as dead." Proposes four-day certificates fetched daily, and states the
slack principle explicitly: a failed fetch is harmless because the previous certificate
is still valid, giving everyone time to fix the problem.
Carry forwardThe design goal of a short lifetime is not a short lifetime. It is a renewal you can afford to have fail.
ieee-security.org/TC/W2SP/2012/papers/w2sp12-final9.pdf
Paper
ISRG2019-11
Let's Encrypt: An Automated Certificate Authority to Encrypt the Entire Web (CCS '19)
The measurement that settles the argument. Renewal lapse affected about 20% of trusted
certificates before Let's Encrypt; 2.9% of its own renewals happen after expiry. It also
documents that the 90-day lifetime was chosen as a behavioural forcing function rather
than for a cryptographic reason.
Carry forwardCite this when someone argues that shorter lifetimes will cause more outages. The measured effect is the opposite, conditional on automation.
abetterinternet.org/documents/letsencryptCCS2019.pdf
Eng blog
Let's Encrypt2025-01-22
Ending Support for Expiration Notification Emails
Josh Aas explaining why the last human-legible warning was switched off on 4 June 2025:
automation is now widespread, the email addresses are a privacy liability, and the
service cost tens of thousands of dollars a year. Consistent with the strategy, and it
removes a backstop from exactly the estates that still need one.
Carry forwardIf a person in your organisation was the alerting mechanism, you no longer have one. Build the renewal-failure alert.
letsencrypt.org/2025/01/22/ending-expiration-emails
Eng blog
Let's Encrypt2025-02-20
We Issued Our First Six Day Cert
The operational guidance that comes with a six-day lifetime, and the clearest statement
of renewal cadence anywhere: renew "every two to three days", check the renewal API "at
least once per day". That is a slack of two to three attempts, deliberately chosen.
Carry forwardRenewal cadence scales with lifetime. A fixed 30-day-out renewal is meaningless below about 90 days.
letsencrypt.org/2025/02/20/first-short-lived-cert-issued
Eng blog
Uber2023-11-09
Our Journey Adopting SPIFFE/SPIRE at Scale
What internal machine identity looks like at 4,500 services across four clouds and
hundreds of thousands of hosts. The problems are not cryptographic: caching thousands of
identities per host, signing load on the identity server, and false positives in agent
health checks under load.
Carry forwardShortening lifetimes multiplies issuance load. Size the issuance plane as a production service with its own SLO.
uber.com/en-IE/blog/our-journey-adopting-spiffe-spire/
Eng blog
Netflix2015-09-21
Introducing Lemur
Worth reading now precisely because it is eleven years old and describes today's
problem: procurement handled by humans, internal and external authorities with no common
record, and expiry as an availability risk. The tooling changed and the ownership gap
did not.
Carry forwardThe inventory with a named owner per credential is the prerequisite, not the follow-up.
netflixtechblog.com/introducing-lemur-ceae8830f621
Eng blog
Keyfactor2024-04-08
Key takeaways from the 2024 PKI and Digital Trust Report
The only public source of industry-wide rates for this failure, and a vendor survey, so
weigh it accordingly. The figure that matters is not the incident count: it is 2.6 hours
to identify an outage against 2.7 hours to remediate it, which says detection is half the
problem.
Carry forwardHalf your mean time to recovery in this class is spent working out that a credential expired. That half is the cheapest to remove.
keyfactor.com/blog/key-takeaways-from-the-2024-pki-digital-trust-report/
Eng blog
TechCrunch2018-12-07
Here's what caused yesterday's O2 and SoftBank outages
Reported rather than first-party, and included because it is the largest published blast
radius in this class and because the absence of an Ericsson engineering account is itself
the finding. Roughly 32 million UK subscribers and a nationwide Japanese outage, from
certificates inside shipped vendor software.
Carry forwardAsk vendors for the expiry dates embedded in what they ship you. Nobody in your organisation owns those and nobody can see them.
techcrunch.com/2018/12/07/heres-what-caused-yesterdays-o2-and-softbank-outages
Source
cert-managercurrent
cert-manager FAQ: renewal timing and terminal failure
The defaults most Kubernetes estates are running without having read them. Renewal at
two thirds of the actual duration, terminal failure visible only as a condition and a
status.lastFailureTime field, and retries backing off from one hour to a cap
of thirty-two.
Carry forwardAlert on lastFailureTime being set. It is the earliest signal you get and nothing surfaces it by default.
cert-manager.io/docs/faq/
Source
cert-manager2023-09-28
Issue #6378: renewal fails during issuer downtime and keeps failing
A renewal agent that cannot recover from a transient fault in its own dependency. The
backoff widens from one hour to four to more than thirteen, no new request is generated,
and a human has to delete the failed object. Open since September 2023.
Carry forwardA stuck agent is indistinguishable from a healthy idle one on a days-to-expiry dashboard. Watch the agent, not only the clock.
github.com/cert-manager/cert-manager/issues/6378
Source
cert-manager2024-11 to 2026-05
PR #7399: add a renewal window, closed unmerged after eighteen months
Operators asked for cron-scheduled renewal windows so rotations happen outside business
hours. The pull request sat from November 2024 and was closed in May 2026 with "This
feature is now implemented in #8258". The replacement merged in March 2026. The gap is
the argument: continuous renewal is the protocol direction and change windows are the
operational reality.
Carry forwardIf you need a renewal window, the underlying problem is that rotation is user-visible. Fix the reload path instead.
github.com/cert-manager/cert-manager/pull/7399
Source
cert-manager2026-03-23
PR #8258: certificate renewal policy and windows, merged
The redesign that did land, "to allow disabling or scheduling of certificate renewals"
using a cron start and a duration. Worth reading next to #7399 as a record of how long a
contested operational feature takes to reach a release when it cuts against the
project's direction.
Carry forwardThe capability now exists upstream. Use it to schedule, not to defer; a window that skips a cycle silently spends your slack.
github.com/cert-manager/cert-manager/pull/8258
Source
Kubernetes SIG Cluster Lifecycle2024-05
cluster-api issue #10522: rotation causes webhook downtime
Measures the cost of the thing everyone assumes is free. The API server trusts the new
CA immediately while the pod waits up to 60 to 90 seconds for the secret to propagate,
producing about 49 seconds of admission failures, roughly every 60 days.
Carry forwardTrust the new issuer before you serve the new credential, and keep the previous one valid until the new one is confirmed in use.
github.com/kubernetes-sigs/cluster-api/issues/10522
Talk
IETF 1212024-11-06
ACME Renewal Information and profiles, working group slides
Aaron Gable of Let's Encrypt presenting draft-ietf-acme-ari-06 to the ACME working
group, alongside the certificate profiles proposal that later let Let's Encrypt offer a
six-day profile at all. Included partly because it is one of very few citable talks on
this topic.
Carry forwardThe design argument for this subject happens in IETF drafts and CA/Browser Forum ballots, not on conference stages. Read the ballots.
datatracker.ietf.org/meeting/121/materials/slides-121-acme-ari-and-profiles-00