Observability Platform  ·  View 20 of 25  ·  Operations

Release Pipeline

Config, rules and a 12,000-host agent fleet, released like production software because that is what they are.

Editable source SVG draw.io All views
Source Collector config pipelines, processors Alert & SLO rules per service Budgets & classes from the catalogue Build Render manifests one artefact per region Rule unit tests fixture series in, alert out Gates Sampled input refused rules must read the complete path Cardinality estimate before it reaches production Config schema + replay against a recorded corpus Cost delta preview on the pull request Environments Staging full pipeline, 1% traffic mirror Canary region one region, 30 min soak All regions wave by wave Fleet rollout Agent rollout rings 1% · 10% · 100% of hosts Two conventions in flight no synchronised deploy Rollback to last-known config one action Release Pipeline — Config, Rules and the Agent Fleet Security / platform Application we own Decision point Opportunity failure / alternate An agent rollout is an estate-wide deploy that no service team asked for. It rides rings and is reversible in one action, because the blast radius is every host at once. The red path is a failed canary soak. v 1.0 · owner Reliability Architecture · date 2026-09

Gates that are load-bearing

  • A rule whose condition reads the sampled corpus is refused at submission. The completeness boundary is enforced by a CI gate, not by a convention in a wiki.
  • A cardinality estimate runs before a metric reaches production, which is what turns the rejection in view 05 into a review comment instead of an incident.
  • A cost delta is previewed on the pull request, so a retention or label change is priced at the moment it is proposed.

Decisions

  • Agent rollouts ride rings — 1%, 10%, 100% of hosts — because an agent release is an estate-wide deploy that no service team asked for.
  • Two instrumentation conventions may be in flight at once, so an estate-wide convention change never requires a synchronised deploy of 900 services.

Risks

  • Replay against a recorded corpus only catches what the corpus contains. A pipeline change that mishandles a signal shape nobody recorded passes every gate.
  • Rolling back config is one action; rolling back an agent across 12,000 hosts is not, however good the rings are.
◀  Deployment Architecture All views Observing the Observability Platform  ▶
Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views20-release-pipeline