Observability Platform  ·  View 10 of 25  ·  Structure

Platform Services

Three planes that scale on different things and fail independently.

Editable source SVG draw.io All views
Read plane — scales with readers, not with ingest Query services Query frontend admission + result cache Metric querier Mimir Trace querier Tempo Log & group-by engine ClickHouse Cold rehydration Athena, ≤ 15 min Consoles Amazon Managed Grafana SAML from the IdP Generated service views from the catalogue Alert evaluation plane — separate node group, separate scaling, separate blast radius Rule evaluator 30 s period SLO burn evaluator multi-window Absence detector expected emission Alert dispatcher context attached Control plane — the only non-derived state in the platform Authoring and policy Config as code Git + Argo CD Policy & rule store Aurora PostgreSQL Catalogue sync owner, class, budget Economics Usage meter bytes, series, queries Cost ledger per team, daily Paging platform (out of scope) Service catalogue all reads firing alert ownership last-known rules Platform Services — Query, Evaluation and Control Application we own Interface / broker Security / platform Data store External / third party synchronous event / async batch The ruler reads the metric store directly, never the query frontend, so a query-tier outage cannot stop a page. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • The alert evaluation plane runs on its own node groups and reads the metric store directly, never through the query frontend. A query-tier outage therefore cannot stop a page — this is the separation that lets alerting carry 99.99% while interactive query carries 99.9%.
  • The control plane is the only non-derived state. It is held twice: in Aurora for serving and in Git for recovery.
  • Everything on the read plane fails static on a control-plane outage: last-known rules, budgets and classes keep working, and only changes are impossible.

Numbers

  • Rule evaluation period 30 s by default, with the full rule set evaluating within one period at p99; missed evaluations ≤ 0.01% per month, each one recorded.
  • Control plane ≥ 99.5% monthly — deliberately the weakest target in the platform, because nothing serving depends on it.

Risks

  • Isolating the ruler costs real capacity that sits idle most of the time. It is the first thing a cost review will propose merging back, and merging it back silently removes the independence the availability targets were written against.