API Key and Token Service  ·  View 22 of 22  ·  Act 7 · Assurance

Failure Modes

Eight named failure classes, what each costs, and the price of the chosen response.

Editable source SVG draw.io All views
What fails Blast radius Response Cost of the choice Control plane down Spanner or management API No issuance or scope change Verification unaffected Customers cannot rotate Revocation channel down Pub/Sub or the poll path Overlays stop advancing Fail closed at 120 s Writes refused platform-wide Region partitioned One region loses the feed That region only Staleness on every response Read-only degradation KMS unreachable Pepper fetch fails Issuance halts Verify on cached version Pepper rotation blocked Poison config Bad vocabulary or policy Every point, at once Refuse, keep last good Config spread to reconcile Store exfiltrated Full database copy taken Digests, no usable secrets Rotate the pepper Dual-version verify window Cold restart storm Fleet restarts together Empty projections Snapshot warm-up, gated admission Slower rollouts False leak positive Anomaly misfires A live customer integration Opt-in only, reversible, attributed Slower containment by default Failure Modes — what is assumed to break, and what happens then Every row trades availability for revocation correctness, in that direction. The fourth column is the price, stated rather than hidden. v 1.0 · owner Security Platform Architecture · date 2026-09

The direction of every trade

  • Every row trades availability for revocation correctness, in that direction, and the fourth column states what that costs rather than hiding it.
  • The hardest row is the revocation channel: failing closed at 120 s refuses writes platform-wide during an outage the customer did not cause. It is still the right side of the trade, and it is the row most likely to be argued about in review.

The rows that surprise

  • Store exfiltration is a pepper rotation, not a mass re-issuance — the single largest practical benefit of keeping the pepper outside the database.
  • A false leak positive is listed as a failure mode with a cost, because an over-eager containment is a real outage and pretending otherwise is how anomaly detection ends up on by default.

Not on this page

  • Ordinary node and zone loss, which is handled by the deployment topology in view 16 and needs no distinct response.