Architecture One-Pager

Observability Platform · Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views

Aggregates are complete and instances are sampled. Nothing is ever counted from the sampled corpus, and every result says which side of that line it came from.

Nine hundred services across three regions produce 25 million metric samples, 1.2 million log lines and 2.5 million spans per second, and an engineer paged at 03:14 for a service they have never opened has ten minutes to name the failing component. Keeping all of it is unaffordable; sampling all of it makes every count an estimate; and the failure nobody notices — telemetry that silently stopped arriving — looks exactly like health. The platform must also survive the incidents it exists to explain, because its own load rises five- to ten-fold at precisely those moments, and it runs on the estate it is watching.

Telemetry leaves an application over localhost to a node agent that owns batching, spooling and shedding, so no application ever waits for or buffers on behalf of the platform. A gateway tier applies resource identity from the credential the workload proved, normalises, redacts before anything is durable, and enforces per-service cardinality budgets as an admission decision. There the pipeline splits: the metric path is reduced by aggregation and stays complete; the trace and log paths are reduced by selection — tail sampling with a 45-second hold and declared log classes — and are deliberately incomplete. Both write through one durable buffer, which makes every store replayable and is the only component carrying a real recovery objective. Object storage is the retention substrate for all three signals, so retention is a lifecycle rule and query capacity scales without moving data. Alert evaluation runs on its own capacity, reads the complete path directly, and cannot be stopped by a dashboard stampede. A separate account in a separate region watches the whole thing and pages when it goes quiet.

What it is, and what it is not

• Complete where it is counted, sampled where it is read — not one corpus at one sampling rate, with every count quietly an estimate.
• Cardinality admitted at the gateway, with the attribute named — not cardinality discovered from the metric store's memory pressure an hour later.
• A gap shown as a gap, and absence alerted as its own condition — not an empty panel that reads like a healthy service.
• A shed order declared before the incident and published — not dropping whatever the pipeline happens to drop when it saturates.
• Alert evaluation on its own capacity, reading only complete data — not paging that shares a query tier with every engineer refreshing a dashboard.
• Cost attributed daily to the team whose behaviour caused it — not a platform bill that is nobody's problem until it is everybody's.
• A witness in another account that pages when we go quiet — not a platform that is the only judge of whether it can still see.

The decisions that are the architecture

1. Split the planes at the gateway (ADR-01) — The metric path is reduced by aggregation and never sampled; traces and logs are reduced by selection and are explicitly an evidence corpus. They are joined only through an index that carries pointers and never quantities.
2. Enforce the boundary in CI (ADR-03) — An alert rule whose condition reads the sampled corpus is rejected when it is submitted, not warned about at runtime. The architecture's central claim is checked by a machine on every change.
3. Admit cardinality, do not discover it (ADR-06) — Budgets are enforced at the gateway before anything is written, a label is dropped before a series is rejected so the aggregate survives, and every action names the attribute and the owning team.
4. Attribution follows proof (ADR-05) — Resource identity comes from the credential the workload proved, never from the payload. Every budget, bill and access decision downstream is only as trustworthy as that one step.
5. One buffer, one replay point (ADR-11) — A single durable stream between admission and every writer turns a storage outage into lag rather than loss, and leaves exactly one component in the platform carrying a real RPO.
6. Shed in a published order, never the metric path (ADR-10) — Debug logs, then access logs, then trace rate. Counting stays true through the storm, and every shed volume is attributed to a team and shown where the data is missing.
7. Paging does not share fate with dashboards (ADR-17) — Alert evaluation runs on its own node groups and reads the metric store directly, which is what lets it carry 99.99% while interactive query carries 99.9%.
8. Buy a witness that cannot go blind with you (ADR-22) — A minimal self-telemetry and dead-man path in a separate account and region, on managed services, sufficient to alert that the platform itself has stopped seeing.

Why it holds up over time

Mimir, Tempo, ClickHouse and the collector will all be replaced. The stack in this document is the least durable thing in it. What should still be right in ten years is the set of boundaries the components sit inside.

• The counting/evidence distinction is not a technology choice. Whatever stores the data, a number derived from selected records still needs the selection rate, and a platform that cannot state it is still quietly wrong. The boundary survives every migration underneath it.
• Admission at the edge stays the right place for a limit. Enforce in the library and you cannot change it; enforce in the store and it fails as an outage. The middle is where you can still explain the limit to the person who caused it, and that does not move.
• The load will always correlate with the incident. Every future observability platform will face a storm at exactly the wrong moment. Having the shed order written down and published before the storm is the transferable part, not the mechanism that implements it.
• Separation of storage and compute keeps getting more right. Object storage economics improve relative to attached disk every year, so a design that makes retention a lifecycle rule and query a scaling decision gets cheaper over time rather than more awkward.
• Attribution outlives the billing model. Whether cost is a chargeback, a quota or a central absorption in five years, the record of which team's behaviour caused which byte is the thing that makes any of those workable.
• The circular dependency never goes away. An observability platform running on the estate it observes cannot be its own only witness. The answer is always somewhere with a different fate, whatever that somewhere is called next decade.
• Fail static ages well. As data planes grow relative to control planes, keeping the store that holds policy out of the path that serves traffic becomes more valuable, not less.
• Honesty about completeness is becoming table stakes. The pressure on every data system to state provenance and limits is increasing. A platform that already labels every result with its coverage is ahead of that, not retrofitting to it.

Non-functional targets

The requirement's targets, how this design meets each one, and the view where the mechanism is drawn. Every figure is a stated assumption for an estate of this shape, to be replaced by measurement before build.

Scope

In scope

• Collection of metrics, logs and traces from 900 services, 12,000 hosts and the managed AWS estate
• Admission: resource identity, normalisation, redaction and per-service cardinality budgets
• Reduction: metric aggregation, head and tail sampling, log class reduction, and the published shed order
• Storage: one durable buffer, three signal stores, the exemplar index, and five retention tiers on object storage
• Query: one surface across the three signals, the aggregate-to-instance pivot, and completeness on every result
• Alert and SLO evaluation on an isolated plane, including absence detection, up to the point of emission
• Tenancy, onboarding from the service catalogue, cost attribution and per-team scorecards
• Self-telemetry and meta-alerting from outside the platform's own failure domain

Explicitly out of scope

• Incident response: routing, escalation, on-call schedules and acknowledgement, which belong to the paging platform
• Product analytics, which belongs in the warehouse and makes an operational store slow and expensive
• The regulatory audit log, whose custody requirements this platform deliberately does not offer
• Automated remediation or any action taken on the estate as a result of what is observed
• Business-level dashboards and reporting built on telemetry, which are consumers rather than platform

Prove it before production depends on it

Ten weeks, one region, forty services across six teams and about 900 hosts, with the aim of falsifying the four assumptions that would change the architecture rather than demonstrating that the pipeline runs.

1. Measure the tail sampler's memory at the real offered span rate and at 5× it, and establish whether 45 seconds covers p99.9 of in-house request duration or whether the window has to grow
2. Run the cardinality admission processor in shadow mode against real production traffic for four weeks and count how many services would have been degraded or rejected, and on which attributes
3. Time the aggregate-to-trace-to-log pivot end to end with engineers who have not seen the service, and record how often the exemplar is missing
4. Replay a recorded incident's telemetry at 5× and confirm the shed order behaves as published, that counts stay true, and that the gap markers reach the query results
5. Measure the real cost per signal against the $22 per host per month target, split by ingest, retention tier, query and evaluation
6. Kill the query tier during a live investigation and confirm that alert evaluation is unaffected and that missed evaluations remain zero

• Pass: tail sampler memory within the provisioned envelope at 5× burst, cardinality shadow-mode degradations under 2% of services, pivot success rate above 90% on error traces, counts provably unchanged through a replayed storm, and cost per host within 25% of target.
• Redesign the sampler: if the 45-second window does not cover p99.9 of request duration, or memory at 5× exceeds what is affordable, move to storage-based selection (ADR-08's deferred option) before building anything else on top of it.
• Reopen ADR-13: if cold rehydration is requested fewer than three times in ten weeks and no audit need appears, downsample-and-discard becomes the default and the 400-day raw log tier is dropped from the MVP.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 24 records across 8 areas, each with the alternatives that lost and what the choice costs.