API Key and Token Service  ·  View 18 of 22  ·  Act 6 · Operations

Observability

Five concerns across five columns, including the one row that catches a failure nothing else would.

Editable source SVG draw.io All views
Issuance Propagation Verification Containment Cost Golden signal Issuance p99 ≤ 400 ms Propagation p99 ≤ 10 s Verify p99 ≤ 3 ms Leak-to-action p99 ≤ 60 s Cost per verification Correctness Escalation attempts blocked Feed sequence gaps Staleness distribution Auto-actions reversed Durable reads on hot path Proof, not assertion Synthetic issue every 60 s Revocation canary issue → revoke → re-verify Known-bad key probe Leak drill, monthly Per-tenant cost report Silence detection No issuance in 15 min Heartbeat missing the alarm that matters Verification volume drop Scanner feed quiet 24 h Watched from In-region Separate project In-process counters Separate project Billing export Observability — what is watched, and where it is watched from The revocation canary is the one metric that would catch a silent propagation failure. It runs from a project this platform's operators cannot deploy to. v 1.0 · owner Security Platform Architecture · date 2026-09

Proof, not assertion

  • The revocation canary issues a credential, verifies it, revokes it and re-verifies it, continuously, in production. Propagation p99 is a measurement from that loop, not a number in this document.
  • It runs from a separate project that this platform's operators cannot deploy to, because a platform cannot be the only judge of whether it can still revoke.

Silence detection

  • A stopped revocation feed, a scanner integration that quietly died, a verification volume that dropped — each looks like health to a monitor that only watches for errors. Each gets its own alarm on absence.
  • The heartbeat is the load-bearing one: its silence is what starts the 120-second staleness clock.

Assumptions

  • Verification telemetry sampled at 1% on success and 100% on denial, so observability cost scales with problems rather than with traffic.
  • Durable-store reads on the verification path are counted and alerted on as an exception, not budgeted as a steady-state cost.