Every request your estate serves begins with the question "who is asking, and may they?", and one central remote service answers it. Between 2020 and 2025 that service failed, in public, at Google, Microsoft, AWS, Auth0 and Cloudflare. This guide reconstructs what kept working during those outages and why, so you can decide what in your own design keeps answering when sign-in cannot.
Authentication is the one dependency shared by everything else, which is why its outages are the only ones that take down products that share nothing else.
State the problem without naming a product: a distributed estate must answer "who is asking, and may they?" on every request, the answer comes from one logically central service, and that service is made of the same databases, caches, queues and deploy pipelines as everything else. So it fails at the same base rate as everything else, while being the one dependency nothing can route around. Google's December 2020 report says its User ID Service lost its quota to an automated tool and authentication failed for about 47 minutes; Gmail and YouTube fell over not because they broke but because they could no longer tell who anyone was.
The surprise in this research is not that identity providers fail. It is where the survivability actually comes from. Microsoft publishes the number that explains it: over 90 percent of Entra authentications are reauthentications of existing sessions, not new sign-ins. The systems that ride out an identity outage are not the ones with a more available identity provider. They are the ones that exploit that skew: they split identity into an issuance side that mints new sessions and a validation side that re-checks existing ones, and they arrange for validation to keep answering from material it already holds, keys, tickets, tokens, replicated decision data, while issuance is down. That split is not new. The 1988 Kerberos paper already had it: a service verifies a ticket with its own secret key, no call to the authentication server, and the ticket lifetime bounds how stale that trust can get.
Scope, stated up front. This guide covers surviving the unavailability of the identity plane: user sign-in, service-to-service credentials, and request-time authorization checks. It does not cover responding to identity-provider compromise (a different problem with opposite instincts: there you want everything to stop trusting cached material), does not cover choosing a federation protocol, and does not cover certificate expiry during normal operation, which is its own failure domain. Fail-open versus fail-closed is covered in depth, because the evidence says that choice, made per enforcement point and usually by default rather than by decision, is where the real risk sits.
The same shape recurs across Google, Microsoft, Netflix, AWS, SPIFFE and the authorization systems: issuance is treated as a control plane that may be absent, and validation is pushed to the caller with everything it needs held locally.
Netflix is the cleanest published example of the shape. Its edge, described in the 2021 Edge Authentication post, cracks open whatever credential arrives (cookie, JWT, MSL token) and mints a short-lived internal identity object, the Passport, integrity-protected with an HMAC, which every downstream service verifies locally. The ratio that matters comes from the engineer who built it: in the QCon SF 2019 talk, Satyajit Thadeshwar reports that only about 5 percent of requests need a remote call from the edge proxy, and that "this split of 95% and 5%... helped a lot with the resiliency of Zuul". Validation at the caller, issuance behind it, and the remote dependency confined to the minority path: that is the reference architecture in one sentence.
AWS applies the same split to its own identity system and names it. The
IAM
security blog divides IAM into a control plane (create and modify principals and
policies) and a data plane, "the authorization system that is used at scale to grant
access", with STS in the data plane. The
Builders'
Library names the property the data plane must have: static stability, meaning
everything the system was already doing continues to work when a dependency is impaired,
even if nothing new can be configured. SPIFFE/SPIRE, the open workload-identity stack,
implements the same idea and, uniquely in this corpus, exposes it as a number: the agent's
availability_target,
"the minimum amount of time desired to gracefully handle SPIRE Server or Agent downtime",
must be at least 24 hours, and the agent rotates certificates early so that a workload
always holds at least that much remaining lifetime.
Authorization data follows the same pattern as authentication keys: replicate it to where the question is asked. Google's Zanzibar paper is blunt about why the central check cannot be allowed to be slow or absent: "authorization checks are often in the critical path of user interactions", and the system holds 99.999 percent availability over three years at more than 10 million checks per second by serving from caches and replicated state rather than a single store. Airbnb's Himeji, per InfoQ's 2021 write-up, fronts its store with a sharded, replicated in-memory cache; the current job posting puts it above 8 million checks per second. Neither system survives its database being down forever; both survive it being down for the length of a cache lifetime, which is the same trade the token systems make.
The one divergence point worth calling out: a separate backup issuance path. Microsoft, after the 2020 and 2021 outages, built one. The Entra backup authentication system is a second service that syncs session metadata while the primary is healthy and reissues tokens during an outage, runs active-active handling a share of traffic continuously so it cannot rot cold, and deliberately serves only reauthentications: same app, same device, used within the last three days. No other organisation in this corpus has published such a system. The honest reading is that a backup issuer is an identity-provider-operator's move; everyone else gets most of the same protection from token lifetime and caches, which cost nothing to operate.
A short-lived, integrity-protected credential minted once at the boundary and verified locally by every hop. The 1988 Kerberos ticket, Netflix's Passport and SPIFFE's SVID are the same design three decades apart.
Runs this way at: Netflix, SPIFFE/SPIRE adopters
Signing keys and policy distributed to validators while everything is healthy, so validation has no runtime read against the issuer. The failure mode moves to rotation, which is exactly where Azure AD broke in March 2021.
Runs this way at: Microsoft, AWS (as static stability)
Authorization answers served from caches and replicas co-located with the check, so the central store's outage budget equals the cache lifetime, not zero.
Runs this way at: Google (Zanzibar), Airbnb (Himeji)
Four forks recur in every system studied. Two of them are really the same number viewed from different sides: how long you let held trust stay valid.
failure_mode_allow: false by default (ext_authz proto); Kubernetes reversed its webhook default from Ignore to Fail at GA (KEP-492 vs v1 docs); Duo moved its Windows logon default from fail-open to fail-closed in 4.3.0 (release notes) after the fail-open default was exploited.| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Where validation happens | At the caller, against held material | Per-request call to the issuer | Issuer availability would gate every request | Revocation must land in seconds | Netflix talk, 2019 |
| Credential lifetime | Explicit outage budget (SPIRE: at least 24h when set) | Reflexively short TTLs everywhere | Lifetime minus rotation point is exactly how long you survive issuer downtime | Theft of a long-lived credential costs more than an outage | SPIRE docs, 2026 |
| Failure mode at the boundary | Fail closed (Envoy default, K8s v1 default, Duo 4.3.0 default) | Fail open | Fail-open paths get triggered deliberately, not just by outages | Check is supplementary AND audited AND lockout blocks recovery | CISA, 2022 |
| Backup issuance path | Microsoft: separate active-active reauth-only service | Cold standby issuer | A cold second issuer rots; Entra's handles live traffic continuously | You are not an IdP operator: lifetime and caches buy most of the same cover for free | Entra blog, 2024 |
| Authorization data placement | Replicated/cached next to the check | Single consistent store queried per check | Check volume (10M+ QPS at Google) and critical-path latency | Checks are rare and strongly consistent denial is a legal requirement | Zanzibar, 2019 |
| Policy evaluated mid-outage | Pre-declared degraded policy (Entra resilience defaults) | Deciding during the incident | The backup path only honours choices recorded before the outage | Never; Microsoft states defaults "must be reenabled before an outage occurs" | Entra CA docs, 2024 |
One absence is worth recording: I could not find a single closed-unmerged pull request in Envoy, Gatekeeper or Kubernetes that tried to flip a fail-open/fail-closed default and was rejected. The argument happens in issues and documentation instead, and in Gatekeeper's case the fail-closed request (issue #248) has stayed open since 2019 while the default remains Ignore. Read that as a signal: projects treat the failure-mode default as too consequential to change casually, so whatever default you inherit is the one your estate will run.
Seven published incidents and advisories, in four classes. None of them is an exotic identity failure; every one is ordinary infrastructure failing underneath the one service that gates everything else.
failure_mode_allow: false.Audit is expected to pick up any slack in enforcement. Gatekeeper documentation, on why its admission webhook defaults to fail-open; the honest version of a trade most teams make silently.
The planning quantity this corpus keeps producing is slack: credential lifetime minus rotation point, which is how long validation keeps answering after issuance stops.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Share of authentications that are reauthentications | >90% | Microsoft Entra | Vendor-stated; the basis of the backup system's reauth-only design | 2024 | Entra docs |
| Backup-auth eligibility window | 3 days | Microsoft Entra | Same app, same device; no new sessions, no guests | 2024 | Entra docs |
| Edge requests needing no remote identity call | 95% | Netflix | Measured; stated as the basis of edge-proxy resilience | 2019 | QCon talk |
| Global sign-in outage duration | 47 min | Measured, 03:46 to 04:33 Pacific | 2020-12 | incident #20013 | |
| Auth-error window, key-rotation incident | ~14 h | Microsoft Azure AD | Measured; mitigated at 2h, long tail from caches | 2021-03 | PIR LN01-P8Z |
| Cognito elevated sign-in failures | ~9 h | AWS us-east-1 | Measured downstream, 05:15 to 14:28 PST | 2020-11 | customer status |
| Identity logins failed during KV outage | 100% for 2h28m | Cloudflare Access | Measured; KV itself failed 90.22% of requests | 2025-06 | postmortem |
| IdP vendor login outage | 4 h 20 min | Auth0 | Reported via RCA coverage; database congestion | 2021-04 | Downtime Project |
| Central authz availability | 99.999% | Google Zanzibar | Measured over 3 years, p95 under 10 ms | 2019 | ATC '19 paper |
| Central authz check volume | >10M QPS | Google Zanzibar | Measured; over two trillion ACLs stored | 2019 | ATC '19 paper |
| Authorization checks per second | ~850k then 8M+ | Airbnb Himeji | 850k/s reported 2021 (InfoQ); 8M+/s in a live 2026 job posting | 2021 / 2026 | InfoQ, Airbnb careers |
| Default workload-credential lifetime | 1 h | SPIRE | default_x509_svid_ttl; default rotation at half-life gives ~30 min of slack | 2026 | SPIRE server docs |
| Minimum declared outage budget | 24 h | SPIRE availability_target | Floor the project accepts when the knob is set at all | 2026 | SPIRE agent docs |
| IdP uptime commitment | 99.99% | Okta | Vendor claim/SLA, cell-based active-active architecture | 2022 | Okta whitepaper |
Measured and vendor-claimed figures are mixed above and labelled in the Context column. The Entra 90 percent figure and Okta's 99.99 percent are vendor statements with no independent measurement published anywhere I could find. The arithmetic worth doing on your own estate: 99.99 percent still concedes about 52 minutes a year, almost exactly one Google-December-2020 per year, so size your token lifetime to survive at least that. And note the asymmetry the SPIRE defaults encode: a 1 hour credential rotated at half-life leaves roughly 30 minutes of slack, which would not have covered a single incident in the table above.
Every source behind this page, graded. Filter by kind. The full claim-by-claim ledger ships alongside this file as sources.md.
The authoritative account of the December 14 sign-in outage: a quota-system migration under-reported the User ID Service's usage as zero, automation cut its quota, and authentication failed globally for about 47 minutes.
Key-rotation automation ignored a key's "retain" flag and deleted it; token validation broke across Microsoft and third-party apps. Rollback took two hours; the tail ran past fourteen as caches re-converged.
The canonical hidden-dependency incident: Cognito buffered analytics writes to a failed Kinesis until its own web servers choked. The fix made sign-in tolerate the dependency's absence rather than removing it.
Workers KV's storage sat partly on a third-party cloud; when it failed, Access failed 100 percent of identity logins and WARP could not register devices. Unusually precise per-product impact percentages.
The live statement during the event, naming "an outage of a 3rd party service that is a key dependency" and listing Access and WARP among affected products.
The downstream view: a SaaS vendor's own users locked out for the roughly nine-hour window of elevated Cognito failures, with the vendor able to do nothing but wait.
State actors re-enrolled a dormant account, then redirected MFA calls to localhost so the fail-open default admitted them. The advisory explicitly warns that fail-open "can happen to any MFA implementation".
The failure-mode decision reduced to one boolean, shipped fail-closed: "When set to true, the filter will accept the client request even if communication with the authorization service has failed... Defaults to false."
A non-UTF-8 header produced an invalid message to the auth service; with fail-open set, those requests were allowed through. The workaround was the default: fail closed.
The same bypass class one year later, via invalid gRPC requests. Two independent CVEs against one flag is a pattern, not an accident.
Open since 2019: the request that Gatekeeper fail closed, including the observation that a freshly started webhook pod that has not yet loaded constraints is itself a fail-open window.
Field report of the fail-closed lockout: a stopped cluster with failurePolicy Fail could not come back because starting the webhook required API operations the absent webhook was blocking.
The only system in this corpus that names the outage budget as a config key: minimum 24 hours, enforced by rotating credentials early so workloads always hold that much remaining lifetime. Default credential TTL elsewhere in the docs is one hour.
A rare written argument for a fail-open default: single-instance webhook availability, the audit controller as compensating control, and the node-deletion circular dependency that fail-closed creates, with the emergency escape documented.
Documents that v1beta1 defaulted failurePolicy to Ignore; the v1 API shipped with the default reversed to Fail. A recorded default-flip on exactly the question this guide is about, in the opposite direction from Duo's.
States the availability bar and hits it: checks "often in the critical path of user interactions", 99.999 percent availability over three years, more than 10 million checks per second over two trillion ACLs, served from replicated and cached state.
The original issuance/validation split: a ticket sealed under the service's own key carries identity, lifetime and a session key, so the service verifies with no call to the authentication server, and lifetime bounds the staleness. Summary copy linked; the original is in the USENIX Winter 1988 proceedings.
The transcript carries the load-bearing ratio: only 5 percent of edge requests need a remote call, "helped a lot with the resiliency of Zuul". Slides detail the HMAC-protected Passport minted at the edge.
A careful outside reading of Auth0's April 2021 RCA: large queries plus cache misses congested the database behind login for 4 hours 20 minutes. The IdP failed as a database application, because that is what an IdP is.
How token handling was pulled out of every microservice and centralised at the edge, with a cryptographically verifiable Passport propagated instead of raw credentials.
A vendor narrating a customer breach through its own product's fail-open setting: a RADIUS proxy configured "safe" (fail open) admitted an attacker during a connectivity loss to Duo's service.
The structural fix after June 12: KV's data moved onto Cloudflare's own infrastructure with third-party clouds demoted to redundancy, removing the single point the login path had been standing on.
Coverage connecting the September 2020 outage (latent defect in the Safe Deployment Process backend, roughly 21:25 to 00:23 UTC) to the March 2021 key deletion, including Microsoft's own admission that the promised SDP hardening was still mid-rollout.
A Zanzibar-shaped check service fronted by a sharded, replicated in-memory cache; roughly 850 thousand checks per second in 2021, with the current job posting claiming over 8 million. Availability comes from the cache tier, not the store.
The only published backup issuer: syncs session metadata while healthy, reissues tokens for existing sessions during outages (same app, same device, three-day window), runs active-active, and explicitly does not serve new sessions or guests.
The degraded-mode policy contract: conditions that cannot be evaluated in real time are either blocked or evaluated from session-start data, the choice is recorded per policy in advance, and enabling it mid-outage does nothing. Sign-in frequency controls reduce what the backup can serve.
The default reversal in the shipped product: fail-open through 4.2.2, fail-closed from 4.3.0, with exe upgrades forcing the new default. Paired with PSA-2019-001, where fail-closed systems could silently behave fail-open through a defect.
What an IdP operator commits to: self-contained cells, active-active-active across availability zones, 99.99 percent uptime. A vendor claim, and still about 52 minutes a year of conceded downtime for the thing in front of all your logins.
Seven rungs from a toy IdP to a measured, drilled estate. The line from toy to real is crossed at rung four, where you start measuring instead of believing.
Run a local OIDC provider (Keycloak or Dex), one web app that validates JWTs against a cached JWKS, and sign in once. Then stop the provider and keep clicking.
Done when: you can state precisely which requests still succeed with the IdP dead, and why. Teaches: issuance and validation are separate systems wearing one name.
Add a second service that calls the provider's token-introspection endpoint per request instead of validating locally. Kill the provider again and compare the two services' behaviour.
Done when: the introspection service fails while the local-validation one serves. Teaches: a remote check inherits the issuer's availability, request by request.
Issue 5-minute and 8-hour tokens to different test users, take the provider down for an hour, and chart surviving traffic against outage time for each cohort.
Done when: you can draw the survival curve and point at where each cohort dies. Teaches: token lifetime is simultaneously your outage budget and your revocation lag; one number, two prices.
Put a gateway (Envoy ext_authz or equivalent) in front, configure fail-closed on the login route and fail-open on a low-risk route, and emit a metric on every fail-open pass-through, copying Envoy's failure_mode_allowed stat.
Done when: inducing a checker outage trips an alert within a minute, naming the route. Teaches: fail-open without a detective control is silent; the metric is the control.
Block egress to your real identity provider from a staging environment for one hour, during business hours, and inventory everything that breaks: cron jobs, service accounts, CI, webhooks, the admin console you planned to fix things with.
Done when: a written list of fail-closed surprises exists and each has an owner. Teaches: the hidden callers of the identity plane, which no architecture diagram shows.
Pre-provision a break-glass credential in a vault with its own trust root, write the runbook, and prove an operator can reach the control plane with SSO fully down. Test it quarterly; an untested break-glass is a prop.
Done when: a timed drill gets an operator from cold start to privileged access without touching the IdP. Teaches: recovery must not authenticate through the thing being recovered.
For every credential class in production (user sessions, service tokens, workload certificates), compute lifetime minus rotation point, publish it as an explicit availability target the way SPIRE does, and alert when the fleet minimum drops below it.
Done when: a dashboard shows the minimum surviving-hours number per credential class, and it has an agreed floor. Teaches: outage tolerance is a measurable property you set, not a hope.
The queries that actually found this material, grouped by what they surface. The incident set will have grown by the time you read this.
<idp name> outage postmortem "root cause" authenticationstatus.<vendor>.com incident history authentication"authentication errors" "post-incident report" key rotationsite:cisa.gov "fail open" MFA advisoryfailure_mode_allow CVE envoy ext_authzfailurePolicy "fail closed" gatekeeper issue"fail open" OR "fail closed" authentication outage "we"duo "fail open" default windows logon"backup authentication service" site:learn.microsoft.com"statically stable" authentication data plane control plane"availability_target" spire SVID rotationnetflix "edge authentication" passport identity propagationzanzibar authorization "critical path" availability paper"authorization checks per second" engineering blog cache<company> careers authorization "checks per second"Research for this guide ran in a sandbox whose network policy allowed direct fetches only to GitHub; GitHub-hosted sources were fetched directly, and all others were retrieved and quoted through search-engine retrieval of the live pages on the check date. Quotes come from that retrieved text, never from memory.