Evidence ledger
28 sources
Checked 15 Oct 2026
Evidence ledger
One row per claim in When the identity provider goes down: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Tier grades follow the skill's hierarchy (postmortem > source/adr >
paper/talk > casestudy/blog > vendor).
Access note for auditors. This guide was researched from a sandbox whose egress policy allowed direct fetches only to github.com / raw.githubusercontent.com. GitHub-hosted sources (Envoy proto, Gatekeeper docs and issues, Kubernetes KEP and docs source, SPIRE docs, the Zanzibar paper text via authzed/zanzibar-annotated) were fetched directly in this session. All other sources were retrieved and quoted through search-engine retrieval of the live pages during this session; quotes below are copied from that retrieved text, not from memory. Where a quote could only be obtained second-hand, the row says so.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Google Cloud incident #20013 (Google Workspace/Cloud auth outage) | postmortem | 2020-12-18 | 2026-10-15 | https://status.cloud.google.com/incident/zall/20013 | The Dec 14 2020 outage was caused by the User ID Service being migrated to a new quota system while parts of the old system reported its usage as zero; automated tools then cut its quota. | "the User ID Service was being moved to a new quota system… parts of the old system kept reporting its usage as zero… automated tools cut the service's quota" (as retrieved; dashboard incident text) | |
| 2 | Same incident, duration and blast radius | postmortem | 2020-12-18 | 2026-10-15 | https://status.cloud.google.com/incident/zall/20013 | Account authentication failed for ~47 minutes and most authenticated traffic errored. | authentication failed "between 3:46 and 4:33 a.m. Pacific"; safeguards missed the case "because the reported load was zero" | |
| 3 | BleepingComputer | Google explains the cause of the recent YouTube, Gmail outage | blog | 2020-12 | 2026-10-15 | https://www.bleepingcomputer.com/news/google/google-explains-the-cause-of-the-recent-youtube-gmail-outage | Corroboration of Google's statement on duration and scope. | quotes Google: requests could not be verified and "most authenticated traffic returned errors" |
| 4 | Microsoft | Azure status history PIR, tracking LN01-P8Z (authentication errors across Microsoft services) | postmortem | 2021-03 | 2026-10-15 | https://s3.documentcloud.org/documents/20515443/authentication-errors-across-multiple-microsoft-services-tracking-id-ln01-p8z.pdf | The Mar 15 2021 Azure AD outage was caused by key-rotation automation deleting a signing key that had been marked "retain" for a cross-cloud migration; tokens signed with it stopped being trusted. | automation "incorrectly ignored" the retain state and removed the key; key metadata rolled back at 21:05 UTC |
| 5 | Microsoft | Same PIR, timeline | postmortem | 2021-03 | 2026-10-15 | https://s3.documentcloud.org/documents/20515443/authentication-errors-across-multiple-microsoft-services-tracking-id-ln01-p8z.pdf | Impact window ~19:00 UTC Mar 15 to 09:25 UTC Mar 16 (long tail from caches and downstream services). | "Between 19:00 UTC (approx) on March 15, 2021, and 09:25 UTC on March 16, 2021" |
| 6 | RCP Magazine | What happened: Azure AD outage (Sept 2020 retrospective in Mar 2021 coverage) | blog | 2021-03-16 | 2026-10-15 | https://rcpmag.com/articles/2021/03/16/what-happened-azure-ad-outage.aspx | The Sept 28 2020 Azure AD outage ran ~21:25 UTC to 00:23 UTC and was attributed to a latent defect in the Azure AD backend Safe Deployment Process system. | "a latent code defect in the Azure Active Directory backend's Safe Deployment Process system"; restored "by 00:23 UTC" |
| 7 | AWS | Summary of the Amazon Kinesis event in US-EAST-1, November 25 2020 | postmortem | 2020-11 | 2026-10-15 | https://aws.amazon.com/message/11201/ | Cognito's sign-in errors were collateral of a Kinesis outage; Cognito used Kinesis for analytics, buffered on error, and the buffers filled. | "The team was working in parallel on a change to Cognito to reduce the dependency on Kinesis. At 10:15 AM PST, deployment of this change began"; "by 2:18 PM PST Cognito was operating normally" |
| 8 | AWS | Same summary, the structural fix | postmortem | 2020-11 | 2026-10-15 | https://aws.amazon.com/message/11201/ | The fix made Cognito statically tolerant of the dependency's absence rather than removing the dependency. | "modified the Cognito webservers so that they can sustain Kinesis API errors" without exhausting buffers |
| 9 | Sparkcentral (Cognito customer) | Status page incident: login issues on PROD | postmortem | 2020-11-25 | 2026-10-15 | https://status.sparkcentral.com/incidents/ntf10pw9rnnx | Downstream measured window of Cognito failures (~9 hours of elevated login failures for a customer that outsourced sign-in). | "increased API failure rates for Cognito User Pools, and Identity Pools in the US-EAST-1 Region between 5:15 AM and 2:28 PM PST" |
| 10 | Cloudflare | Cloudflare service outage June 12, 2025 | postmortem | 2025-06 | 2026-10-15 | https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/ | Workers KV's storage backend was partly on a third-party cloud that failed; KV failed 90.22% of requests; the outage lasted 2h28m. | "failure in the underlying storage infrastructure used by our Workers KV service"; "part of this infrastructure is backed by a third-party cloud provider"; "90.22% of requests failing" |
| 11 | Cloudflare | Same post, identity blast radius | postmortem | 2025-06 | 2026-10-15 | https://blog.cloudflare.com/cloudflare-service-outage-june-12-2025/ | Access failed all identity-based logins; WARP could not register or authenticate new devices because of Access and KV dependencies. | Access "failed 100% of identity based logins for all application types including Self-Hosted, SaaS and Infrastructure"; WARP "impacted due to core dependencies on Access and Workers KV, which is required for device registration and authentication" |
| 12 | Cloudflare | Cloudflare status incident 25r9t0vz99rp | postmortem | 2025-06-12 | 2026-10-15 | https://www.cloudflarestatus.com/incidents/25r9t0vz99rp | Live statement of the third-party dependency and affected products (Access, WARP among them). | "Workers KV went offline due to an outage of a 3rd party service that is a key dependency" |
| 13 | Cloudflare | Redesigning Workers KV for increased availability and faster performance | blog | 2025-08-08 | 2026-10-15 | https://blog.cloudflare.com/rearchitecting-workers-kv-for-redundancy/ | The structural fix: data moved onto Cloudflare's own infrastructure, third-party clouds demoted to redundancy. | post describes re-architecting "KV's redundant storage backend to remove single points of failure" |
| 14 | Microsoft | Microsoft Entra: backup authentication system | vendor | 2024 (doc, updated) | 2026-10-15 | https://learn.microsoft.com/en-us/entra/architecture/backup-authentication-system | Microsoft operates a separate backup issuance service; it reissues tokens for existing sessions only, and most authentications are reauthentications. | "Reauthentications of existing sessions make up over 90% of Entra authentications"; introduced 2021; no new sessions, no guest users |
| 15 | Microsoft | Same doc, the eligibility window | vendor | 2024 (doc, updated) | 2026-10-15 | https://learn.microsoft.com/en-us/entra/architecture/backup-authentication-system | The backup path covers a user only if they used the same app on the same device recently. | user "must have authenticated in the last three days using the same app on the same device" |
| 16 | Microsoft | Conditional Access: resilience defaults | vendor | 2024 (doc, updated) | 2026-10-15 | https://docs.azure.cn/en-us/entra/identity/conditional-access/resilience-defaults | During an outage, policy conditions that cannot be evaluated in real time are either blocked or evaluated from session-start data; the choice is pre-declared per policy and cannot be flipped mid-outage. | resilience defaults "must be reenabled before an outage occurs"; sign-in frequency and authentication-methods policies "reduce resilience" |
| 17 | Microsoft | Microsoft Entra resilience update (Entra blog) | vendor | 2024 | 2026-10-15 | https://techcommunity.microsoft.com/blog/microsoft-entra-blog/-/4094703 | The backup system runs active-active and serves a share of traffic continuously rather than sitting cold. | runs "active-active alongside the primary service, handling a portion of traffic all the time" |
| 18 | Netflix | Edge Authentication and Token-Agnostic Identity Propagation | blog | 2021-02 | 2026-10-15 | https://netflixtechblog.com/edge-authentication-and-token-agnostic-identity-propagation-514e47e0b602 | Netflix centralised token handling at the edge and propagates a verifiable identity object (Passport) instead of letting every service call an identity service. | moved auth "to the edge of the network, managed by a set of centralized services, and a single team"; "cryptographically-verifiable token-agnostic identity object" |
| 19 | Netflix / InfoQ | User & Device Identity for Microservices @ Netflix Scale (QCon SF 2019, transcript) | talk | 2020 (transcript) | 2026-10-15 | https://www.infoq.com/presentations/netflix-user-identity/ | 95% of edge requests resolve identity locally; only 5% need a remote call, and that ratio is what protects the edge proxy. | "This split of 95% and 5% where for only 5% of the requests we need to make remote calls from Zuul, helped a lot with the resiliency of Zuul." (transcript; a precise video timestamp could not be verified from this session) |
| 20 | Netflix / QCon | Slide deck for the same talk | talk | 2019-11 | 2026-10-15 | https://archive.qconsf.com/system/files/presentation-slides/qconsf2019-satyajit-thadeshwar-user-device-identity-for-microservices-netflix-scale.pdf | Passport is created at the edge per request and integrity-protected. | Passport: "identity structure created at the edge for each request", "integrity protected by HMAC" |
| 21 | Zanzibar: Google's Consistent, Global Authorization System (USENIX ATC '19) | paper | 2019-07 | 2026-10-15 | https://www.usenix.org/conference/atc19/presentation/pang | Authorization sits in the critical path of nearly every user interaction, which is what sets its availability bar. | "authorization checks are often in the critical path of user interactions" (paper text fetched via the authzed/zanzibar-annotated mirror) | |
| 22 | Same paper, the achieved bar | paper | 2019-07 | 2026-10-15 | https://www.usenix.org/conference/atc19/presentation/pang | The availability and scale a central decision point must hit to be tolerable. | "maintained 95th-percentile latency of less than 10 milliseconds and availability of greater than 99.999% over 3 years"; "more than 10 million client queries per second"; "more than two trillion ACLs" | |
| 23 | Steiner, Neuman, Schiller (MIT) | Kerberos: An Authentication Service for Open Network Systems (USENIX Winter 1988); summary copy | paper | 1988 | 2026-10-15 | https://pages.cs.wisc.edu/~sschang/OS-Qual/security/kerberos.htm | The split this guide describes is the original design: a service validates a ticket with its own secret key, with no call to the authentication server at request time, and the ticket lifetime bounds the exposure. | ticket sealed under the service's key carries names, address, issue time, lifetime, session key; "the receiving service decrypts the ticket using its secret key" (mechanism corroborated across the summary and Heimdal's whatis.texi) |
| 24 | Envoy | ext_authz filter API, failure_mode_allow |
source | current (fetched) | 2026-10-15 | https://github.com/envoyproxy/envoy/blob/main/api/envoy/extensions/filters/http/ext_authz/v3/ext_authz.proto | The failure-mode choice is a single boolean at every enforcement point, and the shipped default is fail closed. | "When set to true, the filter will accept the client request even if communication with the authorization service has failed… Defaults to false." |
| 25 | Envoy | CVE-2023-27488 / GHSA-9g5w-hqr3-w2ph | source | 2023-04 | 2026-10-15 | https://github.com/envoyproxy/envoy/security/advisories/GHSA-9g5w-hqr3-w2ph | Fail-open was converted into a deliberate bypass: a non-UTF-8 header made the check "fail", and failing open admitted the request. | advisory: requests were allowed through when the invalid protobuf message reached the auth service with failure_mode_allow: true; workaround "set failure_mode_allow: false" (quoted via NVD: https://nvd.nist.gov/vuln/detail/CVE-2023-27488) |
| 26 | Envoy | CVE-2024-23324 / GHSA-gq3v-vvhj-96j6 | source | 2024-02 | 2026-10-15 | https://github.com/envoyproxy/envoy/security/advisories/GHSA-gq3v-vvhj-96j6 | The same class recurred a year later with invalid gRPC requests; fixed in 1.29.1/1.28.1/1.27.3/1.26.7. | downstream clients "could send invalid gRPC requests to ext_authz and bypass its checks when failure_mode_allow was true" |
| 27 | Open Policy Agent | Gatekeeper docs: Failing Closed | adr | current (fetched) | 2026-10-15 | https://open-policy-agent.github.io/gatekeeper/website/docs/failing-closed/ | A recorded design argument for defaulting fail-open, with a named compensating control and a named deadlock. | "Currently Gatekeeper is defaulting to using failurePolicy: Ignore… Audit is expected to pick up any slack in enforcement"; node-deletion example: "This circular dependency will need to be broken before the cluster's control plane can recover." |
| 28 | Open Policy Agent | Gatekeeper issue #248: Fail Closed | source | 2019 | 2026-10-15 | https://github.com/open-policy-agent/gatekeeper/issues/248 | The fail-closed ask has been open for years; a fresh webhook pod that has not loaded constraints is itself a fail-open window. | issue asks that a newly started pod not be "overly permissive" while it loads state |
| 29 | Open Policy Agent | Gatekeeper issue #3183 | source | 2023 | 2026-10-15 | https://github.com/open-policy-agent/gatekeeper/issues/3183 | Field report of the lockout: a stopped cluster could not start again with failurePolicy: Fail. |
cluster startup failure reported with "FailurePolicy as Fail" |
| 30 | Open Policy Agent | Gatekeeper issue #733 | source | 2020 | 2026-10-15 | https://github.com/open-policy-agent/gatekeeper/issues/733 | Fail-closed on the namespace-label webhook blocked namespace creation. | "failurePolicy been set to Fail" caused namespace creation failures until changed to Ignore |
| 31 | Kubernetes | KEP-492: Graduate Admission Webhooks to GA | adr | 2019-01-27 | 2026-10-15 | https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/492-admission-webhooks | At v1beta1 the API's own default was fail-open. | "FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Ignore." |
| 32 | Kubernetes | Dynamic Admission Control reference (docs source, fetched) | source | current | 2026-10-15 | https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ | The project reversed the default at GA: v1 defaults to fail closed. | "The default failurePolicy for an admission webhooks is Fail." |
| 33 | SPIFFE/SPIRE | spire-agent configuration reference (fetched) | source | current | 2026-10-15 | https://github.com/spiffe/spire/blob/main/doc/spire_agent.md | SPIRE names the outage budget explicitly: availability_target, minimum 24h, drives early rotation. |
"The minimum amount of time desired to gracefully handle SPIRE Server or Agent downtime… If set, must be at least 24h"; "the agent will rotate an X509 SVID when its remaining lifetime reaches the availability_target" |
| 34 | SPIFFE/SPIRE | spire-server configuration reference (fetched) | source | current | 2026-10-15 | https://github.com/spiffe/spire/blob/main/doc/spire_server.md | The default credential lifetime, hence default slack, is one hour. | "default_x509_svid_ttl … 1h" |
| 35 | CISA / FBI | AA22-074A: Russian state-sponsored actors gain network access by exploiting default MFA protocols and PrintNightmare | postmortem | 2022-03-15 | 2026-10-15 | https://www.cisa.gov/sites/default/files/publications/AA22-074A_Russian_State-Sponsored_Cyber_Actors_Gain_Network_Access_by_Exploiting_Default_MFA_and_PrintNightmare.pdf | Fail-open was triggered deliberately: actors redirected the MFA agent's calls to localhost, and the fail-open default then waved them through; a dormant un-enrolled account allowed device re-enrollment. | actors "redirected Duo MFA calls to localhost"; advisory urges configurations that guard against "fail open" and re-enrollment scenarios |
| 36 | Cisco Duo | Failing open: a lesson in attention to configurations | blog | 2022 | 2026-10-15 | https://duo.com/blog/failing-open-a-lesson-in-attention-to-configurations | A vendor's own incident account of a customer breach through a fail-open RADIUS proxy during a connectivity loss. | an authentication proxy "set by the customer to fail open" let the attacker through during a connection loss to Duo services |
| 37 | Cisco Duo | DUO-PSA-2019-001 | source | 2019 | 2026-10-15 | https://duo.com/learn/psa/duo-psa-2019-001 | Even fail-closed can silently become fail-open through a defect: versions 4.0.0-4.0.4 could react "fail open" when configured fail closed. | a configuration combination "could cause a system set to fail closed to react with a continuous 'fail open' state when unable to reach Duo's service"; fixed in 4.0.5 |
| 38 | Cisco Duo | Duo for Windows Logon release notes (rdp-notes) | vendor | 2024 | 2026-10-15 | https://duo.com/docs/rdp-notes | The shipped default reversed after exploitation: fail-open through 4.2.2, fail-closed from 4.3.0; exe upgrades force the new default. | "Windows Logon 4.3.0 installers default to fail closed"; exe upgrades "override the previous fail mode selection and default to fail closed" |
| 39 | Auth0 (Okta) | Understanding the Auth0 Challenge Settings Fail Open feature | vendor | current | 2026-10-15 | https://support.auth0.com/center/s/article/understanding-the-auth0-challenge-settings-fail-open-feature | A supplementary check (bot challenge) failing open is sanctioned, but the fail-open path leaves no log event, so the compensating detective control is missing. | fail open lets authentication proceed without the challenge when the service is unreachable, and "no explicit log event is created" in that case |
| 40 | The Downtime Project | Auth0's Seriously Congested Database (podcast episode on Auth0's April 2021 RCA) | talk | 2021 | 2026-10-15 | https://downtimeproject.com/podcast/auth0s-seriously-congested-database/ | An IdP vendor's 4h20m login outage came from ordinary database congestion, not identity logic. | "4 hour, 20 minute outage… caused by a combination of several large queries and a series of database cache misses" |
| 41 | Okta | Okta High Availability Architecture whitepaper | vendor | 2022-09 | 2026-10-15 | https://www.okta.com/sites/default/files/2022-09/Okta%20High%20Availability%20Architecture_Whitepaper.pdf | What an IdP operator commits to: cell-based isolation, active-active-active across zones, 99.99% uptime target (vendor claim). | cells "contain all of the infrastructure required to operate"; architecture supports "99.99% uptime" |
| 42 | AWS | Static stability using Availability Zones (Amazon Builders' Library) | vendor | 2019 | 2026-10-15 | https://aws.amazon.com/builders-library/static-stability-using-availability-zones/ | The named pattern this guide keeps finding: keep doing what you were doing when a dependency is impaired. | "everything it was doing before the dependency became impaired continues to work" (retrieved paraphrase of the article's static-stability definition) |
| 43 | AWS | How to monitor and query IAM resources at scale, part 1 (Security Blog) | vendor | 2020 | 2026-10-15 | https://aws.amazon.com/blogs/security/how-to-monitor-and-query-iam-resources-at-scale-part-1 | AWS's own split of identity into control plane (CRUD) and data plane (authorization at scale, including STS). | "The data plane, in contrast, consists of the authorization system that is used at scale to grant access to the broader set of AWS services and resources" |
| 44 | Airbnb / InfoQ | How Airbnb standardised authorization with Himeji (InfoQ coverage of Airbnb's post) | casestudy | 2021-05 | 2026-10-15 | https://www.infoq.com/news/2021/05/airbnb-himeji/ | A central check service made survivable by a sharded replicated in-memory cache in front of the store. | "sharded and replicated in-memory cache"; ~850k entities/sec by March 2021 with 99.999% availability and 12 ms p99 (figures as reported by InfoQ) |
| 45 | Airbnb | Staff Software Engineer, Authorization (careers posting) | vendor | 2026 (live posting) | 2026-10-15 | https://careers.airbnb.com/positions/7782308/ | Current scale of the same system, from the sideways source: the job ad. | Himeji is "the authorization system powering over 8 million authorization checks per second" |