Observability Platform  ·  View 18 of 25  ·  Runtime

Telemetry Storm

The platform's load is correlated with the failures it exists to explain. What it drops, in what order, and who is told.

Editable source SVG draw.io All views
Steady state Onset — 2× Saturation — 5× Beyond the budget Recovery Metric path 25 M samples/s New series from error labels Cardinality budget bites Never shed Complete throughout Trace path 6% retained Error keep rules fire Keep ceiling reached Rate cut, stamped Rate restored Log path 1.2 M lines/s Debug class surges Debug class shed first Access class shed next Backlog drained from buffer Platform response Headroom 1.3× Gateway scales out Buffer absorbs 10 min Shed volume counted per team Replay from retained offset What the reader sees Nothing unusual Freshness lag shown Coverage below 100% Named gap, named class Gap stays on the record Telemetry Storm — Shedding By Class, Not By Luck The order of shedding is declared in advance and is the same every time: debug logs, then access logs, then trace rate. The metric path is not in the order, which is what keeps every count true through the incident. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • The shed order is declared in advance and is the same every time: debug logs, then access logs, then trace rate. The metric path is not in the order at all.
  • Every shed volume is counted and attributed to a team, and appears in the same interface where the data is missing. A gap with a name is recoverable knowledge; a gap without one is a mystery for the next incident.
  • Beyond the burst budget the platform sheds rather than queues without limit, because an unbounded queue turns a 10-minute incident into an hours-long freshness incident.

Numbers (assumptions)

  • Absorb 5× steady state for 10 minutes without loss; shed by class beyond it. Headroom sized at 1.3× steady state for cardinality.
  • Buffer retention 72 h, which is what allows recovery to be a replay from the retained offset rather than a permanent loss.

Risks

  • The shed order is only as good as the class declarations, and classification is the step teams skip (view 05). Shedding the default class first would be shedding at random with extra ceremony.
  • "Absorb the whole storm instead" means sizing ingest and storage for the worst incident rather than the steady state — headroom that is idle almost always. ADR-19 leaves the trade open.
◀  Critical Flow — The Pivot All views Deployment Architecture  ▶
Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views18-telemetry-storm