Observability Platform  ·  View 15 of 25  ·  Runtime

Critical Flow — Sample to Page

Fourteen steps from an OTLP push to a human's phone, entirely on the complete path.

Editable source SVG draw.io All views
Service + OTel SDK Node agent Gateway collector Ingest buffer Metric store Alert evaluation Paging platform 1. OTLP over localhost, fire and forget 2. batched OTLP/gRPC, resource identity attached 3. normalise units, names and clock source 4. redact before anything is written 5. cardinality admission against the service budget 6. label dropped — enforcement event counted 7. aggregate — every observation contributes 8. produce, partitioned by service 9. consume, then commit the offset 10. WAL, then block flush to S3 11. evaluate, 30 s period 12. unsampled series only 13. burn rate, two windows 14. firing alert with context Critical Flow — A Sample Arrives and a Human Is Paged Budget: sample arrival to alert emission p95 ≤ 45 s, p99 ≤ 90 s. The ruler reads the store, never the query frontend — step 11 has no dependency on anything a human is also using. v 1.0 · owner Reliability Architecture · date 2026-09

What the sequence proves

  • Step 11 — the ruler reads the metric store, not the query frontend. Nothing a human is also using sits between a sample and a page.
  • Step 6 — an admission rejection is a message back to the producer and a counted enforcement event, not a silent drop. The data is gone either way; the difference is whether anyone knows.
  • Steps 3 to 7 all happen before the buffer, which is what makes redaction a guarantee rather than a best effort: nothing unredacted is ever written.

Budget

  • Sample arrival to alert emission: p95 ≤ 45 s, p99 ≤ 90 s. Metric ingest to queryable: p95 ≤ 20 s, p99 ≤ 45 s.
  • Rule evaluation period 30 s. The page budget is therefore mostly evaluation period plus write lag, which is where any improvement has to come from.

Risks

  • A consumer-lag incident on the buffer delays every alert in the platform by the lag, and looks to a reader exactly like a quiet estate. This is why freshness is a first-class alerting signal in view 21 rather than a dashboard.
◀  Retention Tiers All views Tail Sampling  ▶
Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views15-metric-path-to-page