Every system delegates "whom do I trust" to a handful of long-lived keys: token signers, host keys, CA roots, the DNSSEC root KSK. This guide reconstructs, from six public incidents and the platforms that rotate on schedule, why the ability to replace those keys routinely is the control that decides whether a compromise is an afternoon or an era. Tomorrow, 11 October 2026, the root of the DNS rotates its key for only the second time in history; this page is the record of what happened the first time, and every time someone could not.
The problem, who has faced it in production, and the one finding to carry into your next design review before reading further.
State the problem without naming a technology: a verifier can only check a signature
against a public key it already holds, so every distributed system ends in a small set of
keys that are trusted by configuration rather than by proof. Those keys live in client
trust stores, resolver config files, known_hosts entries, baked-in JWKS caches
and firmware. Replacing one of them means changing state you do not control, on machines
you have never seen, without breaking the traffic the key protects. That replacement is the
operation this guide is about.
The headline finding, and the reason to read on: the Cyber Safety Review Board reports that Microsoft, as quoted in coverage of the report, "stopped key rotation entirely in 2021, following a major cloud outage linked to the manual rotation process" [CSRB, 2024]. The rotation was paused because rotation was dangerous; the pause left a 2016 consumer signing key alive in 2023, when Storm-0558 used it to forge tokens into 22 organisations' mailboxes. The availability fix created the security incident. Rotation is not hygiene you schedule when convenient. It is a capability, and like any capability it atrophies exactly when exercising it is scary, which is exactly when you will need it.
Scope. This guide covers the keys at the top of a trust chain: how they are held, how a successor is introduced, and what happened when rotation was absent, late or botched. It deliberately does not cover leaf-certificate renewal (the machine-identity dig in this series), the first hour after a leaked secret (the secret-leak dig), or session-token binding (the stolen-session dig). Those are the keys below the anchor; this is the anchor itself.
Across the DNS root, the web PKI, OpenSSH, Entra ID, Kubernetes and Sigstore, the same five-stage shape appears. The variation is in two places: how long the overlap runs, and whether anyone can measure uptake before cutting over.
Strip away the domain detail and every rotation-capable system in the corpus has the same anatomy. A custody layer holds the private key: Microsoft moved its token signing keys into Azure Managed HSM with rotation automated after 2023 [TechTarget, 2024]; the DNS root keeps its HSMs in two safes behind a filmed quorum ceremony, one safe for the HSMs and laptop, one for the smart cards that activate them [APNIC, 2020]; Sigstore spreads its root across five named community keyholders with hardware tokens and published terms [root-signing README]. A publication channel carries public keys to verifiers: a JWKS document, the DNSKEY RRset, an OS trust store, TUF metadata. An overlap window keeps old and new simultaneously valid. A cutover moves signing to the new key. A retirement revokes or deletes the old one.
The stage most architectures miss is the feedback edge: telemetry from verifiers back to the operator before cutover. The DNS root got it only in 2017, when RFC 8145 trust-anchor signalling, "only finalized in April, 2017" per ICANN, revealed that roughly 5% of reporting validators still carried only the old key, and the first rollover was postponed a year on that evidence [ICANN, 2017]. The web PKI has no equivalent: no CA can ask the world's TLS clients which roots they hold, which is why the DST Root CA X3 expiry surprised vendors who run TLS for a living. Where the feedback edge exists, rotation is an engineering project with a progress bar. Where it does not, rotation is a bet, and the overlap window is how much you hedge it.
A token issuer signs continuously, so its key must be online; the fix after Storm-0558 was not to take it offline but to put it in a managed HSM with automated rotation and, per coverage of the Secure Future Initiative, "no potential for human access". A root that signs a handful of times a year can afford air-gapped ceremony custody.
Runs this way at: Microsoft, IANA root zone, Sigstore
RFC 5011 has resolvers accept a new anchor only after seeing it signed by the old one
across a 30-day hold-down. TUF requires new root metadata signed by the previous root
key. OpenSSH servers push replacement host keys to clients in-session via the
hostkeys@openssh.com extension. Same idea, three protocols: continuity is
proven in-band, cheaply, while the old key is still trustworthy.
Specified in: RFC 5011, TUF spec, ssh_config(5)
Microsoft's contract with relying apps is explicit: keys roll "on a periodic basis
and, in an emergency, could be rolled over immediately", so apps must refresh JWKS
every 24 hours and refetch on an unknown kid. Kubernetes shows the
opposite pole: the KEP-740 design record states the API server loads signing keys at
process start, so "rotating keys require a kube-apiserver to restart".
Contracts at: Microsoft, Kubernetes KEP-740
One structural observation the sources force on you: a trust anchor can outlive its own validity wherever the verifier chooses to let it. Let's Encrypt kept certificates working on old Android phones by having IdenTrust cross-sign ISRG Root X1 from the already-expiring DST Root CA X3, which works, in their words, because "Android intentionally does not enforce the expiration dates of trust anchors" [Let's Encrypt, 2020]. Expiry is not a property of the key. It is a property of each verifier's policy about the key, and in a population of verifiers you ship to, every policy exists at once.
Four forks recur in every system in the corpus. For each: what the production systems chose, what they rejected, and the condition that flips the answer.
known_hosts entry at once, and did it anyway "out of an abundance of
caution" because the key had been exposed.UpdateHostKeys in OpenSSH, kid-triggered
JWKS refetch in OIDC. It scales to verifiers nobody can enumerate.| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Cadence | Scheduled, automated | On-demand, manual | Manual rotation caused an outage, so it stopped, so a 2016 key signed until 2023 | CSRB, 2024 |
| Custody | Online HSM for frequent signers, offline quorum for rare ones | One model for all keys | Signing rate and exposure trade directly; the ceremony is the rare key's deploy pipeline | APNIC, 2020; SecurityWeek, 2023 |
| Overlap | Sized to verifier refresh: hours (JWKS) to 21 months (root KSK) to 3 years (cross-sign) | Hard cutover by default | Overlap is free insurance until the old key is hostile | ICANN, 2026; Let's Encrypt, 2020 |
| Introduction | In-band: old key vouches for new | Out-of-band only | Scales to unenumerable verifiers; flips to out-of-band on compromise | RFC 5011; TUF spec |
| Emergency path | Pre-built and rehearsed (GitHub: days from exposure to swap) | Improvised under fire | GitHub published new fingerprints, staged the key, swapped at a stated hour | GitHub, 2023 |
Three failure classes cover every incident in the corpus: the rotation muscle atrophied, the verifier population decided the blast radius, and the rotation apparatus itself failed. Naming the class tells you which control was missing.
The quiet member of this catalogue is Kubernetes, where no public incident exists but the failure shape is documented in the project's own records: issue #22351 reports that "changing the key pair will not automatically regenerate service account tokens", and KEP-740 records that rotation requires an API-server restart. No postmortem names this as a root cause in the public record, which reads less as safety than as the Storm-0558 pattern pre-incident: rotation friction high enough that fleets simply do not rotate. That is an inference, flagged as one; the KEP shipping external signing (alpha in v1.32) suggests the maintainers read it the same way.
The quantities that recur in rotation planning: ages, windows, uptake percentages and custody costs, each with its context and date.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Age of signing key at compromise use | 7 years | Microsoft MSA | Issued 2016, forging tokens 2023; rotation stopped 2021 | 2024 | CSRB |
| Blast radius of one signing key | 22 orgs, 500+ mailboxes | Exchange Online | Forged-token access, measured | 2024 | CSRB |
| Exposure-to-rotation, emergency host key | days (swap 05:00 UTC 2023-03-24) | GitHub.com | Key exposed "this week", replaced at an announced hour | 2023 | GitHub |
| In-band anchor acceptance delay | 30 days minimum | Any RFC 5011 resolver | Add hold-down time; two validated sightings required | 2007 | RFC 5011 |
| Root KSK publication-to-cutover overlap | 21 months | DNS root, KSK-2024 | Published 2025-01-11, sole signer 2026-10-11 | 2026 | ICANN |
| Resolver uptake before second roll | 95%+ | DNS root | Reporting resolvers signalling KSK-2024 | 2026 | ICANN |
| Stuck-resolver share that forced postponement | ~5% (6-8% daily) | DNS root, 2017 | RFC 8145 reporters carrying only KSK-2010 | 2017 | ICANN |
| User failure window after a missed roll | within 48 h | Validating resolvers | SERVFAIL onset per ICANN's rollover guide; 99%+ of validating users expected unaffected | 2018 | ICANN guide |
| Cross-sign lifetime for legacy devices | 3 years | IdenTrust / ISRG | Expired-root cross-sign for pre-7.1.1 Android | 2020 | Let's Encrypt |
| JWKS cache contract | 24 h TTL, 1 h refresh, 5 min floor | Microsoft identity platform | Prescribed relying-party behaviour, refetch on unknown kid | 2026 (living doc) | Microsoft |
| Rogue certificates from one CA breach | 531; 300,000 client IPs | DigiNotar | Issued over 10 days; Iranian users of the rogue Google cert | 2011 | Fox-IT |
| Shared-KMS key custody | $1/key/month + $0.03/10k requests | AWS KMS | Customer-managed key, prorated hourly | 2026 | AWS |
| Dedicated HSM custody | $1.60/hr (~$1,152/mo), x2 for HA | AWS CloudHSM, us-east-1 | Vendor's own worked example | 2026 | AWS |
| Issuance resting on one root's intermediates | ~10M certificates/day | Let's Encrypt | Stated issuance rate, late 2025 | 2025 | ISRG |
The CSRB figures and the RFC values are measured or normative. ICANN's uptake percentages count only resolvers that report RFC 8145 signals, a self-selected modern subset, so the true stuck population is higher than the complement suggests; the IMC 2019 paper flags exactly this telemetry gap. "Far fewer affected than expected" and "overwhelming success" are ICANN's own assessments of its own rollover. AWS prices are vendor list prices, dated on the day checked. The Kubernetes rotation friction is documented behaviour, but its production incident rate is unpublished: nobody knows how many clusters have never rotated their service-account keys.
Every source behind this page, graded. One honesty note: this guide was researched from an environment whose egress allows GitHub raw content and git but refuses direct fetches elsewhere; GitHub-hosted sources were fetched raw (and one PR's merge state verified by git), while all other pages were retrieved through web-search retrieval of their text. The ledger shipped beside this page records which was which, per source.
The definitive account: rotation stopped entirely in 2021 after a rotation-linked outage, no tooling flagged overdue keys, the 2016 key stayed live, and the theft path was never established.
The crash-dump hypothesis, published with specifics, then withdrawn in the March 2024 update: no dump containing the key was found. The investigation outlived its own conclusion.
Private key briefly exposed in a public repository; replaced days later at an announced hour, scoped to one algorithm, with new fingerprints published for verification. The rare public example of a rehearsed emergency anchor swap.
The independent tally of who broke when DST Root CA X3 expired: Palo Alto, Cisco Umbrella, Google Cloud Monitoring, Auth0, Shopify, QuickBooks, Fortinet and more, overwhelmingly via outdated OpenSSL in appliances.
531 rogue certificates in ten days, all eight CA servers compromised, 300,000 Iranian IPs hitting the rogue Google certificate. The report that ended a certificate authority.
A malfunctioning safe mechanism blocked access to ceremony material days before the 40th KSK ceremony. No DNSSEC interruption, but the signing calendar slipped and the safe was drilled open on camera.
A "minimalistic" port of modern alternative-chain verification, opened weeks after the DST Root X3 breakage. Its head commit is absent from master as of 2026-10 while master stays active (verified by git in this session).
The thread that documents the friction: change the service-account key pair and existing tokens are not re-minted; the workaround is deleting token secrets so the controller recreates them.
The protocol extension that "supports graceful key rotation by allowing a server to send replacement public keys" into clients' known_hosts after authentication, fetched raw from the portable tree.
Five named keyholders with dated terms (two already emeritus), signing events as PRs, and machinery that "proposes resigning when signatures are close to expiry". Rotation designed in before the system had users.
The design record that names the constraint flatly: keys load at process start, so "rotating keys require a kube-apiserver to restart", and moves signing behind an RPC so external key managers can rotate without one.
The standards answer to in-band anchor rotation: a new key is accepted only after the old one vouches for it across a hold-down of "30 days or the expiration time of the original TTL... whichever is greater", seen in at least two validated sets.
Roles hold multiple keys with thresholds; compromise below the threshold cannot reach clients; new root metadata is signed by the previous root key so continuity is provable offline.
Brand-new RFC 8145 telemetry showed ~5% of reporting validators carrying only the old key; ICANN postponed the first-ever root rollover days before the date, on data that had not existed six months earlier.
"The result was exceptionally good: There were far fewer Internet users negatively affected by the change... than anyone expected." The operator's own verdict on the first roll, written three weeks after it.
KSK-2024 (key tag 38696) published since 2025-01-11; from 2026-10-11 the root signs only with it; more than 95% of reporting resolvers already signal the new key. The second roll, run as routine.
The operator's-eye view ahead of the second roll: in 2018 Cloudflare "had seen resolvers lose their learned trust in the new key during software upgrades or moves between machines", so publishing early is necessary but not sufficient.
The inside account of the jammed-safe ceremony: two safes, HSMs in one, activation smart cards in the other, and the decision process that ended with a drill and a locksmith under camera.
IdenTrust issued a three-year cross-sign for ISRG Root X1 from the expiring DST Root CA X3, workable because "Android intentionally does not enforce the expiration dates of trust anchors". A verifier quirk, used deliberately, bought years of compatibility.
The project's own warning, seventeen days early, that 1.0.2 "always prefers the untrusted chain" and would follow it to the expired root; the fix is removing the dead root from the trust store, host by host.
"A complete removal from our trusted root program", justified by the six-week nondisclosure. The verifier side of the DigiNotar story: trust-store operators executed the rotation the CA would not.
Chrome distrusted Entrust TLS issuance after "a pattern of compliance failures, unmet improvement commitments" across six years: the slow-motion version of the verifier-side rotation lever, with a dated SCT cutoff instead of an emergency removal.
Five community keyholders generated keys on hardware tokens during a live-streamed ceremony and signed the initial TUF root: a trust anchor born with its rotation protocol already public.
"Frequently issuing ten million certificates per day" as of late 2025. The scale number that explains why root and intermediate hygiene at ISRG is an industrial process, not a ceremony-only affair.
Operator guidance from the resolver-vendor side, including the catch for late joiners: configure close to the roll and "you no longer have 30 days for the hold timer", so the new key goes in by hand.
Independent multi-vantage measurement of the 2018 rollover: "generally did not lead to problems for end users", but "under the hood, a number of issues occurred", including telemetry polluted by non-resolver signals.
Fifteen years of measured key transitions show "measurable gaps relative to prescribed key management processes", and the authors argue noncompliant transitions are inevitable at ecosystem scale.
The TUF paper: start from the assumption that keys will be compromised, then derive thresholds, role separation and offline roots so that rotation, not secrecy, is the survival mechanism.
The operator community briefing between postponement and execution: what the RFC 8145 data showed, what could and could not be inferred from it, and the plan to proceed in October 2018.
The plan as presented to operators while KSK-2017 sat published but idle: the key "ready for operations but not yet in use", and the staged schedule that the September telemetry would later interrupt.
The issuer's explicit contract: keys roll periodically and "in an emergency, could be rolled over immediately"; relying apps must handle it programmatically, cache JWKS for 24 hours, and refetch on an unknown kid.
$1 per customer-managed key per month plus $0.03 per 10,000 requests; a dedicated CloudHSM instance at $1.60/hour, about $1,152 a month, doubled for HA. The three-orders -of-magnitude custody cost spread, from the vendor's own pages.
At the Secure Future Initiative launch: "Key rotation will also be automated allowing high-frequency key replacement with no potential for human access." A year later the progress report lists Entra and MSA signing keys generated, stored and auto-rotated in Azure Managed HSM as complete.
The new root key was generated in April 2024 after the HSM supplier announced its exit; Verisign's observations note the planned three-year rotation cycle had stretched to eight via the pandemic and hardware end-of-support.
Six rungs from a toy issuer to a rotation drill with a measured recovery time. The line from reading to capability crosses at rung four.
Build a JWT issuer that signs with a keypair, publishes a JWKS document with a
kid, and a verifier service that fetches it. Rotate the keypair while the
verifier is handling traffic, keeping the old key in the JWKS during overlap.
Done when: a rotation during steady traffic produces zero verification failures. Teaches: overlap and kid-based selection, the mechanics under every OIDC provider.
Make the verifier cache the JWKS forever, rotate, and watch everything fail. Then
implement the published contract: 24-hour cache, hourly refresh, immediate refetch on
unknown kid with a 5-minute floor, per Microsoft's rollover doc.
Done when: you can state your system's maximum tokens-rejected window after an emergency roll, and demonstrate it. Teaches: the verifier cache, not the issuer, sets your rotation cadence.
Run sshd with two host keys and UpdateHostKeys yes on a client; connect,
then retire the old key and confirm the client carries on without a prompt. Then
simulate the GitHub scenario: delete the server key outright and script the client-side
recovery (ssh-keygen -R plus fingerprint verification against a published
list).
Done when: graceful rotation needs no human, and emergency rotation needs one documented command. Teaches: the difference between the scheduled and the compromised path, on a protocol you already run.
Stand up a TUF repository (the reference implementation or tuf-on-ci, as Sigstore runs) with a 2-of-3 root threshold. Rotate one root keyholder: new root metadata signed by the old quorum. Verify a client accepts the chained root and rejects metadata signed below threshold.
Done when: a client that saw only root v1 safely reaches root v3 offline. Teaches: rotation as a signed state transition the old key authorises.
Declare rung one's current key compromised at an arbitrary moment. Execute: new key live, old key removed from JWKS (no overlap), all long-lived credentials the old key minted invalidated (the Kubernetes #22351 lesson). Measure time to full recovery and count what broke.
Done when: the drill has run twice and the second time was faster; key age and "last rotated" alerts exist. Teaches: rotation as a capability with a number attached, which is what the CSRB found Microsoft lacked.
Before the next scheduled rotation, make verifiers report which key set they hold (a label on a metrics endpoint is enough; RFC 8145 is the production-grade shape). Dashboard the percentage on the new key and write down the threshold that authorises cutover and the back-out that a miss triggers.
Done when: a rotation is postponed or executed because the dashboard said so, like ICANN in 2017. Teaches: the feedback edge that separates a measured rollover from a bet.
The queries that found this material, grouped by what they surface. The vocabulary that unlocks the field: rollover, hold-down, key ceremony, cross-sign, trust anchor, kid.
"stopped key rotation" signing key outage"host key" rotation exposed "out of an abundance of caution""DST Root CA X3" expiration post-mortem OpenSSL 1.0.2"key ceremony" postponed safe OR HSM malfunctionrogue certificates CA breach report "revoked" trust store removalroot KSK rollover postponed RFC 8145 telemetry validatorsRFC 5011 "hold-down" trust anchor rollover"signing key rollover" JWKS "unknown kid" cache"cross-sign" expired root "trust anchors" AndroidKEP "service account" signing key rotation restart site:github.comsigstore root-signing "signing event" keyholder thresholdTUF "root role" rotation "previous root key" specificationUpdateHostKeys "graceful key rotation" hostkeys@openssh.com"KSK rollover" measurement IMC paper resolversDNSSEC "key transitions" measurement longitudinalChrome root program distrust "pattern of compliance failures"