Evidence ledger
34 sources
Checked 10 Oct 2026
Evidence ledger
One row per claim in Rotate the key while nothing is wrong: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Guide: Rotating the key everything trusts: how platforms replace trust anchors, and what happens
to the ones that cannot
Category: security-and-identity · Research date: 2026-10-10.
How these were fetched, stated up front. This session's egress proxy serves
raw.githubusercontent.com (HTTP 200) and the git protocol for public repositories, and refuses
every other host at CONNECT time; github.com web paths answer 403 for every path, existent or
not. The KEP-740 README, the sigstore root-signing README and the OpenSSH ssh_config.5 manual
were fetched raw and quoted from the fetched text; the jruby-openssl pull request's merge state
was verified by cloning the repository and checking whether the PR head commit
(8c04b7102f049ae9311812d3d08c3b338241fb7d) is an ancestor of master (it is not, as of
2026-10-10, while master carries commits through 2026-10-07). Every other host in this ledger
(CISA, Microsoft, GitHub's blog, ICANN, Let's Encrypt, Cloudflare, OpenSSL, Mozilla, Fox-IT's
report at sec.gov, USENIX/ACM mirrors, IETF, NLUUG, DNS-OARC, APNIC, SIDN, AWS, CNCF, Verisign)
was refused by the proxy at fetch time; their content and the quotes below were retrieved through
this session's web-search retrieval, which returns page text, not from memory. verify.mjs
reports those hosts as unreachable warnings rather than failures; the page names this limit.
Where the retrieval returned a report's wording through trade coverage rather than the primary
page, the row says so.
One row per claim. The right-hand column is copied text or a figure, not paraphrase.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | CISA / Cyber Safety Review Board | Review of the Summer 2023 Microsoft Exchange Online Intrusion | postmortem | 2024-04-02 | 2026-10-10 | https://www.cisa.gov/sites/default/files/2025-03/CSRBReviewOfTheSummer2023MEOIntrusion508.pdf | Rotation of the MSA signing keys had stopped years before the theft, so a 2016 key was still valid in 2023. | Board wording as quoted in coverage of the report: Microsoft "stopped key rotation entirely in 2021, following a major cloud outage linked to the manual rotation process"; the stolen key was a 2016 MSA key still active in 2023. |
| 2 | CISA / Cyber Safety Review Board | Review of the Summer 2023 Microsoft Exchange Online Intrusion | postmortem | 2024-04-02 | 2026-10-10 | https://www.cisa.gov/sites/default/files/2025-03/CSRBReviewOfTheSummer2023MEOIntrusion508.pdf | The theft path was never established, and the crash-dump explanation had no supporting evidence. | "Microsoft does not know how or when Storm-0558 obtained the signing key"; "Microsoft has no evidence or logs showing the stolen key's presence in or exfiltration from a crash dump." |
| 3 | CISA / Cyber Safety Review Board | Review of the Summer 2023 Microsoft Exchange Online Intrusion | postmortem | 2024-04-02 | 2026-10-10 | https://www.cisa.gov/sites/default/files/2025-03/CSRBReviewOfTheSummer2023MEOIntrusion508.pdf | Blast radius of one signing key. | Attackers "accessed more than 500 individual mailboxes belonging to 22 organizations" using tokens forged with the stolen key; a validation flaw let the consumer key sign tokens accepted by enterprise systems. |
| 4 | Microsoft MSRC | Results of Major Technical Investigations for Storm-0558 Key Acquisition | postmortem | 2023-09-06 (updated 2024-03-12) | 2026-10-10 | https://msrc.microsoft.com/blog/2023/09/results-of-major-technical-investigations-for-storm-0558-key-acquisition/ | Microsoft's leading hypothesis was a key in a crash dump carried out of the isolated signing environment; the update withdrew the central claim. | Coverage of the post: "a consumer signing system crash in April of 2021 resulted in a snapshot of the crashed process"; the March 2024 update states Microsoft "has not found a crash dump containing the impacted key material." |
| 5 | Microsoft | Analysis of Storm-0558 techniques for unauthorized email access | postmortem | 2023-07-14 | 2026-10-10 | https://www.microsoft.com/en-us/security/blog/2023/07/14/analysis-of-storm-0558-techniques-for-unauthorized-email-access/ | The forged tokens worked against enterprise mail because validation accepted a consumer key outside its domain. | As quoted in coverage: "a validation issue allowed this key to be trusted for signing Azure AD tokens"; MSA and Azure AD keys "are issued and managed from separate systems and should only be valid for their respective systems." |
| 6 | GitHub | We updated our RSA SSH host key | postmortem | 2023-03-24 | 2026-10-10 | https://github.blog/news-insights/company-news/we-updated-our-rsa-ssh-host-key/ | A planned-for host-key rotation executed within days of exposure, scoped to one algorithm. | "This week, we discovered that GitHub.com's RSA SSH private key was briefly exposed in a public GitHub repository"; "At approximately 05:00 UTC on March 24, out of an abundance of caution, we replaced our RSA SSH host key used to secure Git operations for GitHub.com." ECDSA and Ed25519 keys were unchanged. |
| 7 | Scott Helme | Let's Encrypt Root Expiration - Post-Mortem | postmortem | 2021-10 | 2026-10-10 | https://scotthelme.co.uk/lets-encrypt-root-expiration-post-mortem/ | The DST Root CA X3 expiry broke vendors who operate TLS for a living, mostly via outdated chain verifiers. | Helme confirmed failures at "Palo Alto, Bluecoat, Cisco Umbrella, Google Cloud Monitoring, Auth0, Shopify, QuickBooks, and Fortinet" (list as reported from his post by SecurityWeek and others); he attributes most failures to outdated software, mainly OpenSSL. |
| 8 | Fox-IT (report filed at SEC) | DigiNotar public report (interim), "Operation Black Tulip" | postmortem | 2011-09-05 | 2026-10-10 | https://www.sec.gov/Archives/edgar/data/1044777/000119312511241796/dex992.htm | A CA whose issuing keys were compromised had no rotation that could save it; every CA server was reached. | 531 rogue certificates issued 2011-07-10 to 2011-07-20; "at least 300,000 unique IP addresses" in Iran used the rogue Google certificate; all eight servers that managed Certificate Authorities were compromised, including the PKIoverheid CA. |
| 9 | Mozilla | DigiNotar Removal Follow Up | blog | 2011-09-02 | 2026-10-10 | https://blog.mozilla.org/security/2011/09/02/diginotar-removal-follow-up/ | When the issuer cannot be trusted to rotate and disclose, the trust stores rotate it out for good. | "a complete removal from our trusted root program", not a temporary suspension; DigiNotar had "detected and revoked some of the fraudulent certificates 6 weeks ago without notifying Mozilla." |
| 10 | ICANN | Update on the Root KSK Rollover Project | blog | 2017-09-28 | 2026-10-10 | https://www.icann.org/news/blog/update-on-the-root-ksk-roll-project | The first root KSK rollover was postponed because new telemetry showed a population that would fail. | Per ICANN's operational message: "approximately 5% of the total validators and about 6%-8% on any given day report only KSK-2010, and these validators would not resolve correctly after the planned root KSK roll." RFC 8145 telemetry "was only finalized in April, 2017." |
| 11 | ICANN | The recent KSK Rollover: Summary and Next Steps | blog | 2018-10-30 | 2026-10-10 | https://www.icann.org/en/blogs/details/the-recent-ksk-rollover-summary-and-next-steps-30-10-2018-en | The rollover, once executed, caused less damage than anyone had modelled. | "The result was exceptionally good: There were far fewer Internet users negatively affected by the change (called a rollover) than anyone expected." |
| 12 | ICANN | Preparing for the Root Zone KSK Rollover: What You Need to Know | blog | 2026-07-27 | 2026-10-10 | https://www.icann.org/en/blogs/details/preparing-for-the-root-zone-ksk-rollover-what-you-need-to-know-27-07-2026-en | The second rollover is scheduled and measured: KSK-2024 cutover on 2026-10-11. | KSK-2024 (Key Tag 38696) was first published in the root zone on 2025-01-11; from 2026-10-11 the root zone signs only with KSK-2024; "more than 95 percent of reporting resolvers have recognized the new key." |
| 13 | ICANN | 2018 KSK rollover: what to expect (guide, as circulated 2018-08) | vendor | 2018-08 | 2026-10-10 | https://atlarge-lists.icann.org/pipermail/alac-announce/2018-August/004446.html | What failure looks like for an end user whose resolver missed the new key. | "If all of a users' resolvers do not have the new KSK in their trust anchor configuration, the user will start seeing name resolution failures (typically 'server failure' or SERVFAIL errors) at some point within 48 hours of the rollover." ICANN estimated "more than 99% of users whose resolvers are validating will be unaffected." |
| 14 | ICANN | Root Key Signing Key Ceremony Postponed | postmortem | 2020-02-12 | 2026-10-10 | https://www.icann.org/en/blogs/details/root-key-signing-key-ceremony-postponed-12-2-2020-en | The ceremony apparatus itself is an operational dependency that fails. | "one of the security mechanisms that protects the contents of the secure safes was found to be malfunctioning"; the malfunction "does not pose a threat to the material protected inside the safe" but "does trigger a delay in holding the ceremony as scheduled." |
| 15 | APNIC (Geoff Huston publishing Kim Davies' account) | Drilling for the KSK | blog | 2020-03-04 | 2026-10-10 | https://blog.apnic.net/2020/03/04/drilling-for-the-ksk/ | The physical shape of the root's key custody: two safes, HSMs and smart cards. | One safe holds "electronic equipment, including a laptop and two Hardware Security Module (HSM) devices", the other "security deposit boxes, inside of which are the smart cards used to enable/activate the HSM devices"; the jammed safe was drilled open to run the ceremony. |
| 16 | ICANN | ICANN to Generate New DNS Cryptographic Key at April 2024 Ceremony | vendor | 2024-02-28 | 2026-10-10 | https://www.icann.org/en/announcements/details/icann-to-generate-new-dns-cryptographic-key-at-april-2024-ceremony-28-02-2024-en | Even the root's rotation schedule is hostage to its hardware vendor. | New KSK generated at the April 2024 ceremony after the HSM supplier announced it would exit the business; the 2018 rollover "was considered a success." |
| 17 | Verisign | The 2024-2026 Root Zone KSK Rollover: Updates and Observations | blog | 2026 | 2026-10-10 | https://blog.verisign.com/security/2024-2026-root-zone-ksk-rollover-updates-observations/ | The planned three-year rollover cycle slipped to eight years for operational reasons. | The schedule slipped from a three-year cycle, with the COVID-19 pandemic and a 2023 hardware security module end-of-support problem contributing to the delay. |
| 18 | Cloudflare | The keys to the Internet change on October 11. Are you ready? | blog | 2026 | 2026-10-10 | https://blog.cloudflare.com/root-ksk-2024-rollover/ | What operators learned in 2018: resolvers silently lose the new trust anchor between rollovers. | "When we wrote about the first root KSK rollover in 2018, we had seen resolvers lose their learned trust in the new key during software upgrades or moves between machines"; "Validating resolvers need to trust the new key before the switch, as otherwise healthy websites could become unreachable." |
| 19 | IETF (M. StJohns) | RFC 5011: Automated Updates of DNS Security (DNSSEC) Trust Anchors | adr | 2007-09 | 2026-10-10 | https://www.rfc-editor.org/rfc/rfc5011.html | The standards answer to trust-anchor rotation: old key vouches for new, behind a hold-down. | "The add hold-down time is 30 days or the expiration time of the original TTL of the first trust point DNSKEY RRSet that contained the new key, whichever is greater." At least two validated DNSKEY RRSets containing the new key must be seen before acceptance. |
| 20 | Let's Encrypt / ISRG | Extending Android Device Compatibility for Let's Encrypt Certificates | blog | 2020-12-21 | 2026-10-10 | https://letsencrypt.org/2020/12/21/extending-android-compatibility | A trust anchor can keep vouching after its own expiry, where the verifier allows it. | IdenTrust "agreed to issue a 3-year cross-sign for ISRG Root X1 from DST Root CA X3"; this works because "Android intentionally does not enforce the expiration dates of trust anchors"; Android versions before 7.1.1 do not trust ISRG Root X1. |
| 21 | Let's Encrypt / ISRG | DST Root CA X3 Expiration (September 2021) | vendor | 2021 (living doc) | 2026-10-10 | https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ | The announced scope of the expiry: old platforms without the new root fail. | DST Root CA X3 lapsed 2021-09-30; it had been cross-signing ISRG Root X1 "so older devices would trust" Let's Encrypt certificates; older devices without ISRG Root X1 would show certificate warnings. |
| 22 | OpenSSL | Old Let's Encrypt root certificate expiration and OpenSSL 1.0.2 | blog | 2021-09-13 | 2026-10-10 | https://mirror.openssl-library.org/post/2021-09-13-letsencryptrootcertexpire/ | The verifier, not the certificate, decided who broke: 1.0.2 picks the path to the expired root. | OpenSSL 1.0.2 "always prefers the untrusted chain and if that chain contains a path that leads to an expired trusted root certificate (DST Root CA X3), it will be selected for the certificate verification"; the fix is to remove the expired root from the trust store, with "no downside... apart from the need to modify all the potential OpenSSL 1.0.2 TLS client hosts trust stores." |
| 23 | jruby/jruby-openssl | PR #240: try building alternative cert chains during verification | source | opened 2021-10-17 | 2026-10-10 | https://github.com/jruby/jruby-openssl/pull/240 | The client-side fix for expired-anchor path building can sit unmerged for years; verified by git. | PR described as "a very concrete (and 'minimalistic') update to try building alternative chain trust chains with cert verification"; its head commit 8c04b71 (2021-10-17) is not an ancestor of master as of 2026-10-10, while master carries commits through 2026-10-07 (checked by cloning the repository). |
| 24 | Kubernetes | kubernetes/kubernetes issue #22351 | source | 2016-03 | 2026-10-10 | https://github.com/kubernetes/kubernetes/issues/22351 | Rotating the cluster's token-signing key does not re-mint the tokens already issued. | Commenter reports that "changing the key pair will not automatically regenerate service account tokens", asking whether that is intended behavior or a bug; the workaround in the thread is deleting the token secrets so they are recreated. |
| 25 | Kubernetes SIG-Auth | KEP-740: Support external signing of service account tokens | adr | implementable, alpha v1.32 (2024) | 2026-10-10 | https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/740-service-account-external-signing | The design record that names restart-to-rotate as the problem and moves the key out of process. | Fetched raw: "kube-apiserver loads service account keys at process start time and the keys remain the same for the life-time of the process. Thus, rotating keys require a kube-apiserver to restart. External key management eliminates the need to restart." |
| 26 | OpenSSH | ssh_config(5), UpdateHostKeys | source | current manual (option since 6.8, 2015) | 2026-10-10 | https://github.com/openssh/openssh-portable/blob/master/ssh_config.5 | The protocol grew a rotation channel: servers push replacement host keys to clients in-session. | Fetched raw: the option "allows learning alternate hostkeys for a server and supports graceful key rotation by allowing a server to send replacement public keys"; only sshd from OpenSSH 6.8 and greater supports the "hostkeys@openssh.com" extension. |
| 27 | Sigstore | sigstore/root-signing README | source | current (checked 2026-10-10) | 2026-10-10 | https://github.com/sigstore/root-signing | A trust root operated as a repeating, PR-shaped ceremony with term-limited keyholders. | Fetched raw: "All changes to artifacts or metadata require cryptographic signatures from Sigstore keyholders"; signing events fire when a "maintainer proposes a change" or "root-signing proposes resigning when signatures are close to expiry"; the keyholder table shows five holders with dated terms and two emeritus. |
| 28 | CNCF / Sigstore | A New Kind of Trust Root (root key ceremony announcement) | blog | 2021-06-16 | 2026-10-10 | https://www.cncf.io/blog/2021/06/16/a-new-kind-of-trust-root/ | The first ceremony's design: public, threshold, hardware-backed. | Five community keyholders created keys on hardware tokens while streamed live; the keyholders then signed the initial TUF root metadata file. |
| 29 | Samuel, Mathewson, Cappos, Dingledine (CCS 2010) | Survivable Key Compromise in Software Update Systems | paper | 2010 | 2026-10-10 | https://uptane.org/assets/files/samuel_ccs_2010-2e4e7a69695e936b1ad3b0b7010c550a.pdf | The design frame this guide uses: assume key compromise, build rotation in from the start. | The paper "identifies core security principles that allow software update systems to survive key compromise" and designs TUF on them; an attacker compromising fewer than a threshold of role keys cannot compromise clients. |
| 30 | The Update Framework | TUF specification | adr | current (checked 2026-10-10) | 2026-10-10 | https://theupdateframework.io/specification | Rotation is a first-class protocol operation: the root role revokes and replaces keys by threshold signature. | The framework must support "roles with multiple keys" and threshold trust; "if a key were compromised, TUF provides a mechanism for reliably revoking keys: the root role"; new root metadata is signed by the previous root key to prove continuity. |
| 31 | Müller, Thomas, Wessels, Hardaker, Chung, Toorop, van Rijswijk-Deij (IMC 2019) | Roll, Roll, Roll Your Root: A Comprehensive Analysis of the First Ever DNSSEC Root KSK Rollover | paper | 2019-10 | 2026-10-10 | https://par.nsf.gov/servlets/purl/10170347 | Independent measurement of the 2018 rollover: fine for users, messy underneath. | "while the rollover generally did not lead to problems for end users, under the hood, a number of issues occurred that led to worries in the DNS operations community"; ICANN called the event "an overwhelming success" with "no significant outages." IMC 2019 Distinguished Paper. |
| 32 | Osterweil, Fotouhi Tehrani, Schmidt, Wählisch | From the Beginning: Key Transitions in the First 15 Years of DNSSEC (IEEE TNSM 2022; arXiv 2109.08783) | paper | 2021-09 (journal 2022) | 2026-10-10 | https://arxiv.org/abs/2109.08783 | Measured at ecosystem scale, real key transitions diverge from the prescribed procedures. | "Key transitions in the wild show measurable gaps relative to prescribed key management processes"; the authors argue such noncompliant transitions are inevitable in the wild. |
| 33 | Matt Larson (ICANN), DNS-OARC 28 | Update on root KSK rollover (or, We're really doing it this time) | talk | 2018-03-08 | 2026-10-10 | https://indico.dns-oarc.net/event/28/contributions/525/ | The operator community was briefed on the RFC 8145 telemetry that drove delay and restart. | Talk abstract covers ICANN's findings on RFC 8145 resolver trust-anchor data after the October 2017 roll was postponed, and the plan to proceed. |
| 34 | Edward Lewis (ICANN), NLUUG voorjaarsconferentie 2017 | 2017 DNSSEC KSK rollover | talk | 2017-05 | 2026-10-10 | https://nluug.nl/evenementen/nluug/voorjaarsconferentie-2017/talks/edward-lewis-2017-dnssec-ksk-rollover | Conference account of the rollover's design while the new key sat published but unused. | Talk page describes the new key as ready for operations but not yet in use, presenting the rollover plan to operators ahead of the (then-)scheduled 2017 roll. |
| 35 | Microsoft | Signing key rollover in the Microsoft identity platform | vendor | living doc (checked 2026-10-10) | 2026-10-10 | https://learn.microsoft.com/en-us/entra/identity-platform/signing-key-rollover | The issuer's contract with every relying party: keys change routinely and without notice. | Keys roll "on a periodic basis and, in an emergency, could be rolled over immediately"; "All applications that use the Microsoft identity platform should be able to programmatically handle the key rollover process"; recommended JWKS cache TTL 24 hours with hourly refresh, and refresh on an unknown kid. |
| 36 | SecurityWeek (covering Microsoft SFI launch) | After Major Cloud Hacks, Microsoft Unveils 'Secure Future Initiative' | blog | 2023-11-02 | 2026-10-10 | https://www.securityweek.com/after-major-cloud-hacks-microsoft-unveils-secure-future-initiative/ | The structural fix Microsoft announced: HSM-held keys with automated rotation. | Identity signing keys move to "an integrated, hardened Azure HSM and confidential computing infrastructure"; "Key rotation will also be automated allowing high-frequency key replacement with no potential for human access." |
| 37 | TechTarget (covering Microsoft SFI progress report) | Microsoft issues first Secure Future Initiative report | blog | 2024-09 | 2026-10-10 | https://techtarget.com/searchsecurity/news/366611385/Microsoft-issues-first-Secure-Future-Initiative-report | The fix shipped: automatic rotation of token signing keys in managed HSM, a year after the breach. | Entra ID and MSA updates "to generate, store, and automatically rotate access token signing keys using the Azure Managed HSM service" reported completed; the MSA key theft mechanism remained unconfirmed. |
| 38 | Google Chrome Security Team | Sustaining Digital Certificate Security - Entrust Certificate Distrust | blog | 2024-06-27 | 2026-10-10 | https://security.googleblog.com/2024/06/sustaining-digital-certificate-security.html | A trust anchor can be rotated out by its verifiers when the operator will not fix itself. | Google cited "a pattern of compliance failures, unmet improvement commitments, and the absence of tangible, measurable progress in response to publicly disclosed incident reports" over six years; Chrome distrusts Entrust TLS certificates with SCTs after 2024-11-11 (Chrome 131; originally announced as 2024-10-31 / Chrome 127). |
| 39 | AWS | AWS Key Management Service pricing | vendor | current (checked 2026-10-10) | 2026-10-10 | https://aws.amazon.com/kms/pricing/ | What outsourced key custody costs at the commodity end. | $1 per customer-managed key per month, prorated hourly; $0.03 per 10,000 symmetric requests after the free tier; each automatic rotation adds $1/month for the first and second rotations, then stops. |
| 40 | AWS | AWS KMS or AWS CloudHSM? (Security Blog) | vendor | current (checked 2026-10-10) | 2026-10-10 | https://aws.amazon.com/blogs/security/aws-kms-or-aws-cloudhsm-choose-the-right-key-management-solution/ | What dedicated HSM custody costs. | "$1.60 per HSM instance per hour (us-east-1)", about $1,152 per HSM per month; AWS recommends at least two HSMs for high availability. |
| 41 | Let's Encrypt / ISRG | 10 Years of Let's Encrypt Certificates | blog | 2025-12-09 | 2026-10-10 | https://letsencrypt.org/2025/12/09/10-years | Scale of the issuance machine that sits under one root's intermediates. | "as of late 2025 we're frequently issuing ten million certificates per day"; in 2020 the project "reached a billion total certificates issued." |
| 42 | Red Hat | What you need to know about the first-ever DNSSEC root key rollover | blog | 2018 | 2026-10-10 | https://www.redhat.com/en/blog/what-you-need-know-about-first-ever-dnssec-root-key-rollover-october-11-2018 | Late joiners cannot wait out the hold-down; the automated path has a minimum latency. | Advice to operators configuring close to the rollover: "you no longer have 30 days for the hold timer, so you must manually add the new DNSSEC Root Key" to the resolver configuration. |