# Observability Platform

**Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views · 24 architecture decision records**

The internal platform that answers "is it broken, where, and since when" for 900 services on 12,000 hosts across three AWS regions. It collects metrics, logs and traces, reduces them, stores them on object storage, answers questions about them, and evaluates the alerts that page a human — and then stops, handing a firing alert to the paging platform. It ingests 25 million metric samples, 1.2 million log lines and 2.5 million spans per second at steady state, absorbs 5× that for ten minutes, and is built from OpenTelemetry collection, Amazon MSK as a durable buffer, Grafana Mimir, Grafana Tempo and ClickHouse over S3, DynamoDB for the exemplar index, Aurora PostgreSQL and Git for the control plane, and Amazon Managed Grafana for the consoles.

The design rests on one rule: **aggregates are complete and instances are sampled, and nothing is ever counted from the sampled corpus.** The metric path is reduced by aggregation, in which every observation contributes before anything is discarded. The trace and log paths are reduced by selection and are deliberately incomplete. They split at the gateway and are joined only through an index that carries pointers and never quantities.

The decisions that carry the design:

- **The boundary is enforced in the pipeline, and in CI.** An alert rule whose condition reads the sampled corpus is rejected when it is submitted, not warned about at runtime (ADR-03).
- **Cardinality is admitted, not discovered.** Budgets bite at the gateway before anything is written, a label is dropped before a series is rejected so the aggregate survives, and every action names the attribute and the owning team (ADR-06).
- **Attribution follows proof.** Resource identity comes from the credential the workload proved, never from the payload. Every budget, bill and access decision downstream depends on that one step (ADR-05).
- **One buffer, one replay point.** A single durable stream between admission and every writer turns a storage outage into lag rather than loss, and leaves exactly one component carrying a real recovery objective (ADR-11).
- **The shed order is published before the storm.** Debug logs, then access logs, then trace rate. The metric path is not in the order, so counting stays true through the incident (ADR-10).
- **Paging does not share fate with dashboards.** Alert evaluation runs on its own capacity and reads the store directly, which is what lets it carry 99.99% while interactive query carries 99.9% (ADR-17).
- **A gap is a gap.** Absence of signal is alerted as its own condition, every result states its coverage, and nothing is ever interpolated (ADR-16, ADR-18).
- **Buy a witness that cannot go blind with you.** A minimal self-telemetry and dead-man path runs in a separate AWS account and region, because a platform cannot be the only judge of whether it can still see (ADR-22).

The architecture one-pager (including why the design holds up over time) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~12 min) and [docs/decision-record.md](docs/decision-record.md) (~70 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 25 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 24 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..f.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only) |
| `ask.md` | The requirement |

## Rebuild

```bash
bash scripts/build.sh
```

The build needs Node 20+ and the `architecture-diagram-portal` skill checked into this repository (override with `SK=<path>`). Every view passes the geometry check (0 errors, 0 warnings) and the routing check (202 edges, 0 errors, 0 warnings). Every icon resolves (421 nodes, 0 unresolved, 0 weak), 454 icons are embedded across the set, and every relative link resolves.

## Notes on the requirement

- **Stack.** The requirement names Amazon Web Services and this package delivers on it, which is the rotation: AWS has been named once before in this practice and retargeted away from in three other requirements, while the last three use cases ran on Google Cloud and on open-source on-premises stacks. Observability belongs to no particular cloud, so the topic was free to rotate.
- **Managed versus self-managed is decided per component, not as a policy.** Managed services are used where they remove operational load without removing per-tenant control — MSK, DynamoDB, Athena, Managed Grafana, Aurora, and the whole self-telemetry account. Mimir, Tempo and ClickHouse are self-managed because the per-tenant cardinality limits, class-level retention and cost attribution in views 06, 12 and 22 are what this estate is actually buying, and the managed equivalents do not expose them at this granularity (ADR-15).
- **Every number is a stated assumption.** The estate size, ingest rates, freshness and query budgets, retention tiers, recovery objectives and cost targets are invented to be defensible and arguable for a 900-service consumer SaaS, not measured. Three of them are explicitly the first that real data should overturn: the 45-second tail-sampling hold window, the 400-day raw cold log tier, and the $22 per host per month target.
- **The 45-second hold window is the package's largest single bet.** It assumes p99.9 of in-house request duration fits inside it. If it does not, the tail sampler's memory budget and trace freshness both move, and ADR-08's deferred option — storage-based selection — becomes the design. The prototype in the one-pager is built to falsify this first.
- **Cold-with-rehydration is a bet that should be re-tested with data.** Keeping 400 days of raw logs preserves questions nobody anticipated at a cost that grows forever. ADR-13 names the condition that would flip it, and the prototype counts rehydration requests as a pass/fail criterion.
- **The self-telemetry account is a control, not infrastructure.** It is small, it does almost nothing on a good day, and it is the most likely line item to be cut in a cost review. Cutting it leaves the circular dependency in view 01 unanswered, and the package says so in three places rather than one.
- **Two controls depend on a step teams routinely skip.** Log class declaration drives both the reduction policy (ADR-09) and the shed order (ADR-10). If most services accept defaults unchanged, the storm response degenerates towards shedding at random. Classification is therefore part of catalogue onboarding, undeclared services are named on the scorecard, and declaration rate is a prototype pass criterion.
- **Cost enforcement is soft everywhere except cardinality.** Ingest, retention and query are attributed and alerted on, not rejected, because a budget control that drops data during an incident is worse than the overspend. Cardinality is the exception, because it is the one dimension where one team's excess degrades the shared store for everyone (ADR-19).
- **Act 2 is drawn from the requirement, not decorated onto it.** Each of the three journey maps has its trough in a different place — a failed pivot, a cardinality rejection, an unpriceable decision — and each trough has an answer somewhere later in the set. The engineering director's journey is the one with no clean answer, and the set says so.