Observability Platform  ·  View 21 of 25  ·  Operations

Observing the Observability Platform

Five things worth watching across six pipeline stages — and the one row that does not depend on the platform being alive.

Editable source SVG draw.io All views
Collect Admit Buffer Store Query Alert Health Agent up, CPU, RSS Gateway error rate Broker in-sync replicas Writer restarts Frontend 5xx Missed evaluations Freshness & lag Agent spool depth Batch age Consumer lag Ingest to queryable Query p95 by class Sample to page Completeness Hosts not reporting Shed by class, per team Records dropped Gap markers written Coverage on results Rules on stale data Cardinality & cost Series admitted / rejected Bytes per team Active series, tier bytes Query cost per team Rule evaluation cost From outside the estate Canary writes a probe Canary reads it back Dead-man switch Observing the Observability Platform The bottom row is the only one that does not depend on the platform being alive. Four blank cells in it are deliberate: an outside prober should test the round trip, not the internals. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Completeness is a first-class row alongside health and latency. Most platforms measure whether components are up; this one measures how much of the estate's telemetry actually arrived.
  • The outside row is deliberately sparse: an external prober should test the round trip — write a probe, read it back, and alert on our silence — not the internals, which it cannot reach when it matters.

The signal that matters most

  • The dead-man switch. Every other cell alerts on something going up; this one alerts on something that should be there and is not, from a place that cannot fail with us.
  • Hosts-not-reporting and missed-evaluations are the two cells most likely to be quietly disabled for noise, and the two whose absence would make the platform's own failure invisible.

Numbers

  • Self-telemetry path ≥ 99.99% monthly, 90-day retention. Missed rule evaluations ≤ 0.01% per month, each one recorded rather than counted.