Chaos Engineering Platform  ·  View 18 of 21  ·  Operations

Observability

Five signal types across five pipeline stages; two rows are load-bearing.

Editable source SVG draw.io All views
Definition & permit Injection Evaluation Abort & revert Evidence Metrics Definitions rejected Guardrail refusals Active leases Renewal rate Signal freshness p95 Abort latency p99 Revert verify rate Verdict mix Logs Approval decisions Applied vs requested Window evaluations Revert attempts Bundle sealing Traces Dry-run resolve Lease issue to inject Collector to verdict Breach to cleared Write to warehouse Alerts Stale inventory Agent unreachable Evaluator blind Revert unconfirmed Kill switch used Evidence gap Reports Escalations granted Classes exercised Inconclusive share Aborts by cause Untested dependencies Cost per team Observability — Signal Type by Pipeline Stage Application we own Security / platform Data store Risk / gap Two signals are load-bearing: abort latency p99, and the share of runs the platform could not observe. v 1.0 · owner Reliability Architecture · date 2026-09

The two signals that matter

  • Abort latency p99 — the platform's single most important number, because every safety claim in this package is downstream of it.
  • The share of runs the platform could not observe. A rising INCONCLUSIVE rate means the platform is spending fear and returning nothing.

Decisions

  • Revert-verification rate is reported as a count of failures, not a percentage. At 100%-required, a percentage hides the one event that matters.
  • The untested-dependency report is published as a product of this platform, not as an internal metric.
  • Kill-switch use alerts unconditionally, because it should be rare enough that every occurrence is worth a conversation.

Assumptions

  • Signal freshness p95 ≤ 15 s at the evaluator; cost attributed per service and per team.
◀  Release Pipeline All views The Loop That Makes It Worth Doing  ▶
Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records18-observability