Architecture Decision Record

Observability Platform · Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views

The argument these decisions serve is summarised in the Architecture One-Pager.

Twenty-four decisions make up this architecture. Everything else across the twenty-five views is convention, sizing or consequence. Each record states the forcing question, the context that makes it hard, what was decided, how it is realised on AWS, the options weighed, what the choice buys and costs, the conditions that would flip it, why it should still be right in ten years, and the lesson that transfers beyond this platform.

Status of this document. This is a design, not a report on a running system. The rates, latencies, volumes, retentions and costs are the requirement's stated assumptions for a 900-service consumer SaaS estate across three AWS regions, invented to be defensible and arguable rather than absent. They are to be replaced by measured telemetry before build, and several of them — the 45-second tail-sampling hold window, the 400-day cold log tier and the $22 per host per month target — are explicitly the first numbers that real data should overturn.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it is realised on AWS: The concrete mechanism: which service or package, configured how, in which account.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

The completeness boundary: The one decision that defines this architecture: which data is complete, which is a sample, and what is allowed to read each.

• ADR-01 · Aggregates are complete and instances are sampled, and the split is enforced in the pipeline
• ADR-02 · The exemplar index is the only join between the planes, and it carries pointers, never quantities
• ADR-03 · An alert rule that reads the sampled corpus is refused at submission

Collection and admission: How telemetry leaves a process, who says what it is, and where the platform is allowed to say no.

• ADR-04 · A node agent collects; the application never holds a durable buffer and never retries to a remote endpoint
• ADR-05 · Resource identity comes from the credential the workload proved, never from the payload
• ADR-06 · Cardinality is admitted at the gateway, and a label is dropped before a series is rejected
• ADR-07 · Redaction happens before the first write, and every redaction is counted

Reduction and sampling: What is discarded, under what rule, and how the reader finds out.

• ADR-08 · Tail sampling with a 45-second hold, whole traces only, sharded by trace identifier
• ADR-09 · Log reduction is by declared class, not by a global rate
• ADR-10 · The shed order is declared in advance, and the metric path is not in it

Storage and retention: The replay point, the substrate, the tiers, and the one class that outlives the rest.

• ADR-11 · One durable buffer between admission and every writer is the platform's single replay point
• ADR-12 · Object storage is the retention substrate; query capacity scales without moving data
• ADR-13 · Cold data is kept raw and rehydrated, rather than downsampled and discarded
• ADR-14 · The compliance class is separable, with its own retention, its own audience and keys held outside the platform

Query and correlation: How three very different stores answer one investigation, and what every answer must admit about itself.

• ADR-15 · Three signal-specific stores behind one query surface, rather than one engine over one store
• ADR-16 · Every result states its completeness, and a partial answer beats a timeout

Alerting and evaluation: The plane that pages a human, what it is allowed to read, and the failure that looks like health.

• ADR-17 · Alert evaluation runs on its own capacity and reads the store directly
• ADR-18 · Absence of signal is a first-class alert condition, driven by expected emission

Economics and governance: Who pays, who decides, and what the platform is allowed to refuse.

• ADR-19 · Chargeback with soft quotas, and a hard quota only on cardinality
• ADR-20 · Query cost is charged to the reader, and unread data is the headline waste class
• ADR-21 · The service catalogue is the front door, and an unclaimed producer is quarantined

Operations and assurance: Running a platform that depends on the estate it observes, and releasing to 12,000 hosts nobody asked.

• ADR-22 · Self-telemetry runs in a separate account and region, on managed services
• ADR-23 · Config, rules and budgets are code with gates; agent rollouts ride rings
• ADR-24 · Every plane fails static on control-plane loss, and never fails silent

Technology by capability

Every capability on the views, the service or package that provides it, a credible alternative, and the record that justifies the choice. The platform runs on Amazon Web Services with open-source collection and storage; managed services are used where they remove operational load without removing the per-tenant control this estate needs, and self-managed components are used where they do not.

The decisions, and the alternatives that lost

The completeness boundary

The one decision that defines this architecture: which data is complete, which is a sample, and what is allowed to read each.

ADR-01 · Aggregates are complete and instances are sampled, and the split is enforced in the pipeline

Status: Accepted · Shown on views: 08, 02, 16

Which telemetry may be trusted to answer "how many", and which is only evidence of "what happened to this one"?

Context. An observability platform is asked two different questions with the same interface. "What is the error rate" is a counting question, and an answer that is off by the sampling rate is worse than no answer because it looks plausible. "Why did this request fail" is an evidence question, and a corpus containing the interesting cases answers it perfectly well while containing 6% of the traffic. Conflating the two is the standard failure: a team computes a count from a sampled trace corpus, is wrong by a factor of sixteen, and has no way to know. At 25 million samples per second against 2.5 million offered spans per second, keeping everything unsampled is not affordable, and sampling everything makes every count an estimate.

Decision. The metric path is never sampled. It is reduced by aggregation, in which every observation contributes before anything is discarded, and it is the only source for counting, rates, percentiles and alerting. The trace and log paths are reduced by selection and are explicitly incomplete. The two paths separate at the gateway, are written to separate stores, and are joined only through the exemplar index. Every retained trace carries the effective sampling rate that produced it, and the query surface refuses or corrects any count derived from the sampled corpus.

How it is realised on AWS. The OTel Collector gateway runs two pipelines from one receiver set: a metrics pipeline whose last processor is aggregation into Mimir-bound series, and a traces/logs pipeline that exports through a load-balancing exporter to the tail-sampling shards. They share no processor after the admission stage. The effective rate is written as a span attribute at the sampler and carried into Tempo.

What it buys

• Every count, rate and percentile in the platform is true for all of the traffic
• The trace retention rate becomes a cost lever with no correctness consequence
• Alerting can be isolated from the query tier because it reads only one plane

What it costs

• Two pipelines, two reduction policies and an index to join them
• "Group errors by customer id" is answerable only on the sampled corpus, with a coverage caveat
• Every engineer has to learn which side of the line their question is on

Choose differently when. If the estate's traffic were small enough to retain unsampled — roughly two orders of magnitude below this one — the wide-event design wins outright, because the boundary buys nothing and the raw attributes answer more questions.

Why it holds up over time. Storage gets cheaper and sampling gets smarter, but the distinction between a measurement and a specimen does not move. Whatever replaces Mimir and Tempo, a count computed from selected records will still need the selection rate, and a platform that cannot state it will still be quietly wrong.

Lesson. Decide once, in the pipeline, which data is allowed to be counted. Leaving it to the query author means being wrong occasionally and never knowing which time.

ADR-02 · The exemplar index is the only join between the planes, and it carries pointers, never quantities

Status: Accepted · Shown on views: 08, 17

If the complete and sampled planes are separate, how does an engineer get from a spike on a chart to the request behind it?

Context. The boundary in ADR-01 is only tolerable if crossing it is easy. Without a join, an engineer sees a spike and then searches the trace store by hand for something that might correspond to it — which is the manual step that makes people give up and go back to grepping logs. But any join is also a route by which the sampled corpus can leak back into a count: an index that stores how many traces matched would immediately be read as a count, and it would be wrong.

Decision. Latency and error metrics carry exemplars: a trace identifier sampled from the requests behind each aggregate bucket. The exemplar index holds series identifier, timestamp, value and trace identifier, and nothing else. It answers "show me one of these" and cannot answer "how many of these". When no exemplar exists for a bucket, the platform says the evidence was not kept rather than returning an empty result.

How it is realised on AWS. Exemplars are emitted by the SDK and preserved through the collector's aggregation processor, written to DynamoDB keyed on series identifier and time bucket with a 90-day TTL. Grafana's exemplar support renders them as points on the chart; clicking one resolves a Tempo trace by identifier.

What it buys

• The pivot from chart to trace to log is one action and needs no prior query knowledge
• The boundary cannot be crossed accidentally in the counting direction
• The index is small and hot, so the pivot is fast even when the stores are not

What it costs

• A fourth store to operate, with its own availability in the investigation path
• Exemplars must survive aggregation, which constrains the metric pipeline
• When the trace was sampled away the pivot fails, and the honest message is still a disappointment

Choose differently when. If tracing were retained unsampled, the index becomes unnecessary — a direct query on trace attributes serves the same purpose with fewer moving parts.

Why it holds up over time. The exemplar pattern is now in the OpenTelemetry and Prometheus data models rather than in any one product, so the mechanism survives replacing every component underneath it.

Lesson. When you separate two planes for correctness, build the crossing deliberately and make it structurally incapable of carrying what it must not.

ADR-03 · An alert rule that reads the sampled corpus is refused at submission

Status: Accepted · Shown on views: 08, 20, 15

How is the completeness boundary kept true a year after launch, when the people who chose it have moved on?

Context. A boundary that exists only in a design document is a boundary for about two quarters. The specific failure is easy to imagine and hard to notice: a team writes an alert on an error count computed from the log or trace store, it works in staging, and in production it under-counts by the sampling rate — so the alert fires late, or not at all, during exactly the incident it was written for. Nothing about the rule looks wrong.

Decision. Alert and SLO rules may only reference the complete path. A rule whose condition reads the trace or log store is rejected when it is submitted, by a CI gate on the rules repository, with a message naming the boundary. This is a build failure, not a runtime warning: a rule that reaches the evaluator has already been proved to read unsampled data.

How it is realised on AWS. Rules live in Git and are rendered by CI. The gate parses each expression, resolves every datasource reference, and fails the build on any that is not the metric store. Rule unit tests run fixture series through the same evaluator the ruler uses.

What it buys

• The central correctness claim is checked by a machine on every change
• The failure mode is a red build with an explanation rather than a missed page
• Rule authors learn the boundary the first time they cross it

What it costs

• The gate's parser must keep pace with the query language, or it becomes a source of false failures
• Legitimate log-derived alerting — a genuine need for some teams — has to be met by first promoting the signal to a metric

Choose differently when. If the platform ever offered an unsampled log class with a completeness guarantee, rules over that class would be legitimate and the gate would need to distinguish class rather than store.

Why it holds up over time. Query languages change; the property being checked — does this condition read data that is complete — does not, and it is expressible against any of them.

Lesson. An architectural boundary is only real once something automated fails when it is crossed.

Collection and admission

How telemetry leaves a process, who says what it is, and where the platform is allowed to say no.

ADR-04 · A node agent collects; the application never holds a durable buffer and never retries to a remote endpoint

Status: Accepted · Shown on views: 09, 12

Where does telemetry go when it leaves a process, and who is responsible for it not being lost?

Context. The tempting design has each application export directly to a regional endpoint with retry and a local queue. It removes a component, and it makes every application's performance dependent on the observability platform's availability and latency. Worse, it makes the telemetry path a source of application memory growth during a platform incident — the failure mode where the monitoring system takes down the thing it monitors. The estate has 900 services in several languages, so anything asked of the application has to be implemented correctly nine hundred times.

Decision. Applications export over localhost to a node agent and do not retry, buffer durably, or block. The agent owns batching, compression, a bounded local spool, shedding by class and retry to the gateway. The agent bounds its own resource use and sheds its own telemetry before degrading the workload sharing the host. An agent or node loss is a telemetry gap for that host, reported as a gap, and never recovered by asking the application to have kept a copy.

How it is realised on AWS. ADOT Collector as an EKS DaemonSet and as a systemd unit on EC2, receiving OTLP on localhost, scraping host and container metrics without application cooperation, and exporting OTLP/gRPC to the gateway NLB. Resource limits are set on the DaemonSet; the spool is a bounded file-backed queue with class-ordered eviction.

What it buys

• A platform outage is never an application outage
• Batching, compression and shedding are implemented and tuned once
• Infrastructure telemetry is collected from uninstrumented services without their cooperation

What it costs

• An agent fleet of 12,000 to release, version and secure (ADR-23)
• A node loss loses its in-flight telemetry, and the design accepts that rather than hiding it
• A DaemonSet is shared by every pod on the node, so one noisy pod can affect its neighbours' telemetry

Choose differently when. In a small estate with one or two languages and a very high value on losing nothing, direct export with an application-side queue is defensible and simpler.

Why it holds up over time. The agent boundary has outlived several generations of collection software because it is a statement about responsibility, not about a protocol: the application produces, the platform transports.

Lesson. Never ask the thing you are observing to be responsible for the observation surviving.

ADR-05 · Resource identity comes from the credential the workload proved, never from the payload

Status: Accepted · Shown on views: 24, 09

When telemetry arrives claiming to be from checkout-api, what makes that true?

Context. Every downstream property of the platform depends on the answer: ownership, cardinality budgets, cost attribution, retention class, authorisation to read, and who gets paged. If the service name is whatever the payload says, then a misconfigured job can spend another team's budget, a hostile client can pollute a production service's metrics, and the cost ledger is fiction. The convenient default in every collector is to take the resource attributes as given.

Decision. The collection tier overwrites resource identity from the workload's scheduling metadata and from the identity it proved to the gateway. Self-declared service identity in the payload is recorded as a hint and never used for attribution. A credential that identifies a team, an environment or a person rather than a single workload is rejected at ingest. Client telemetry from the untrusted zone can never resolve to an in-house service identity at all.

How it is realised on AWS. EKS Pod Identity and IAM Roles Anywhere issue per-workload roles; the agent signs with SigV4 and the gateway resolves the caller's role ARN to a catalogue entry. Kubernetes downward API supplies namespace, workload and version. The untrusted receiver runs behind API Gateway on a separate listener whose resolved identity is always a client tenant.

What it buys

• Cost, budget and ownership attribution are enforceable rather than cooperative
• A compromised or misconfigured producer cannot impersonate another service
• Access control on the read path has something real to bind to

What it costs

• Every producer needs a workload identity before it can emit — a genuine onboarding dependency
• Ingest depends on STS availability for workloads starting during an incident
• Legacy and third-party producers need a mapped identity or a quarantined tenant

Choose differently when. If the platform were single-tenant with no chargeback and no per-team access control, payload-declared identity would be adequate and much cheaper to run.

Why it holds up over time. Workload identity is moving towards being ambient in every runtime. The rule — attribution follows proof, not assertion — gets easier to honour over time, never harder.

Lesson. Anything you plan to bill, budget or authorise on must be derived from something the sender proved, not something it said.

ADR-06 · Cardinality is admitted at the gateway, and a label is dropped before a series is rejected

Status: Accepted · Shown on views: 09, 22

Where does the platform stop an unbounded label, and what does it do when it finds one?

Context. Cardinality is the dominant cost and failure driver of any metric system, and the classic incident is a deploy that adds a user identifier or a URL with an embedded key to a metric label. The store discovers it as memory pressure, minutes to hours later, by which time the damage is estate-wide. Enforcing in the SDK stops it at the source but puts the limit in a library that cannot be changed without redeploying 900 services. Enforcing at the store is the last line and fails as an outage rather than as a rejection.

Decision. Cardinality budgets are enforced at the gateway as an admission decision, per service and per metric, before anything is buffered or written. When a budget is exceeded the platform first drops the offending label — collapsing the series into a lower-cardinality parent so the measurement survives — and only rejects the series when dropping is not possible. Every action produces an attributed enforcement event naming the specific attribute. A pre-production estimate runs in CI so the common case is caught at review.

How it is realised on AWS. A custom OTel Collector processor evaluates per-service budgets loaded from the control plane, with a label-value ceiling and an unbounded-value detector for identifier-shaped and URL-shaped values. Enforcement events go to ClickHouse and to the owning team's scorecard the same day. Mimir's own per-tenant limits remain configured as a backstop, deliberately above the gateway budget.

What it buys

• One team's bad deploy cannot degrade the platform for everyone else
• The aggregate survives the incident, so dashboards and alerts keep working at reduced detail
• The owning team learns the same day, with the attribute named

What it costs

• The gateway carries budget state and becomes more than a stateless forwarder
• Label-dropping is a silent change of meaning to anyone who does not read the enforcement record
• A legitimately high-cardinality service needs an exception, and exceptions need expiry and review

Choose differently when. If the metric store could absorb unbounded cardinality at predictable cost, this becomes purely a billing question and the enforcement point moves to the ledger.

Why it holds up over time. Every metric system built so far has been cardinality-bound, and the ones that are not shift the same constraint into cost. Admission at the edge stays the right place for it whichever way that goes.

Lesson. Enforce a limit where you can still explain it to the person who caused it, and prefer losing a dimension to losing a measurement.

ADR-07 · Redaction happens before the first write, and every redaction is counted

Status: Accepted · Shown on views: 23, 09

Logs will contain personal data despite policy. Where is that dealt with?

Context. Every organisation has a policy that personal data does not go in logs, and every log store contains some. Detecting it after the fact means it has already been written, replicated, tiered and possibly read — and the remediation is a deletion across every tier rather than a non-event. Detecting it at read time protects the reader and not the store. The only place where a redaction is genuinely preventive is before the record is durable.

Decision. Pattern-based detection and redaction run at the gateway, before anything reaches the buffer. Every redaction is counted and attributed to the emitting service so that redaction volume is a signal about instrumentation quality rather than an invisible safety net. Redaction is explicitly not treated as making the log store safe to open widely: authorisation by data class (ADR-14) is the second control, and the two are argued separately.

How it is realised on AWS. A collector processor applies a maintained pattern set — card numbers, national identifiers, email addresses, bearer tokens, and per-tenant custom patterns — replacing values with a typed placeholder and emitting a counter keyed by service and pattern. Patterns are versioned in the config repository and released through the pipeline in view 20.

What it buys

• The log store's worst-case contents are bounded by what the patterns miss, not by what teams remember
• Redaction volume per service becomes a measurable instrumentation-quality signal
• Subject erasure has a smaller surface to reach

What it costs

• Pattern matching on 1.2 million lines per second is a real CPU cost in the ingest path
• A redacted field is sometimes the field an engineer needed, and there is no recovery
• False confidence is the standing risk: redaction reduces exposure and does not eliminate it

Choose differently when. If the estate could genuinely guarantee structured logging with a typed schema and no free-text bodies, field-level classification would replace pattern matching and be both cheaper and more reliable.

Why it holds up over time. Whatever the regulation and whatever the detection technique, the placement argument holds: the only redaction that prevents rather than remediates is the one that happens before the record is durable.

Lesson. Put a data-protection control where its failure is a miss rather than a cleanup project.

Reduction and sampling

What is discarded, under what rule, and how the reader finds out.

ADR-08 · Tail sampling with a 45-second hold, whole traces only, sharded by trace identifier

Status: Accepted · Shown on views: 16, 09

Which traces are kept, and is that decided before or after the platform knows whether the request was interesting?

Context. Head sampling decides at the first span, cheaply and statelessly, before it knows whether the request errored or was slow — which is exactly what makes a trace worth keeping. Tail sampling decides once the trace is complete, which requires holding every span of every in-flight trace for a window and routing all spans of a trace to the same decision point. At 2.5 million offered spans per second the hold window is a memory budget sized against the worst burst, and the sampler becomes the only memory-bound tier in the ingest path.

Decision. A head-sampled floor provides coarse volume control; tail sampling does the evidence selection. Spans are routed by consistent hash on trace identifier so every span of a trace reaches one shard. The hold window is 45 seconds, assumed to cover p99.9 of in-house request duration. A trace selected by the sampler is kept whole. Keep rules — error status, above the service's p99, debug flag, declared critical routes — are unconditional but carry a per-service keep ceiling, because an unconditional rule is otherwise an unbounded ingest path that fires hardest during an incident.

How it is realised on AWS. Collector gateways export via the load-balancing exporter to a dedicated memory-optimised node group running the tail-sampling processor with policy sets per service. Hold-window occupancy, eviction rate and decision latency are first-class platform metrics; the node group is sized at 5× steady-state burst headroom.

What it buys

• The error trace an engineer needs is present at the moment they look for it
• Retained span volume is roughly 6% of offered, at a known and tunable cost
• Whole-trace retention means a pivot never lands on half an answer

What it costs

• Trace freshness carries 45 s the metric path does not (p95 ≤ 75 s to queryable)
• The sampler must be sized for the worst burst rather than the average
• Shard loss during the hold window loses in-flight decisions, which view 12 names as an accepted volatile store

Choose differently when. If request durations were long-tailed beyond the window — batch or streaming workloads rather than request/response — the hold window becomes untenable and storage-based selection is the right answer.

Why it holds up over time. The head-versus-tail trade is a property of distributed tracing itself, not of any implementation. The constants move; the shape of the decision does not.

Lesson. If you want to keep the interesting cases, you have to wait long enough to know which ones they are, and that wait has a price you must state.

ADR-09 · Log reduction is by declared class, not by a global rate

Status: Accepted · Shown on views: 18, 05

When log volume must come down, which lines go?

Context. A global sampling rate on logs is simple and treats a payment failure and a health-check access line as equally disposable. It also makes the surviving corpus statistically odd in ways nobody tracks. The alternative is to let teams declare what each of their log streams is for, which is correct and depends on a step teams routinely skip — which is itself the risk that has to be designed for rather than assumed away.

Decision. Every service declares its log classes, with platform defaults. Structured error and compliance-relevant classes are retained in full. High-volume debug and access classes are sampled or aggregated. Reduction is applied per class, the reduction ratio per service and class is published, and the shed order under a storm (ADR-10) follows the same classes. A service that declares nothing gets the defaults and is reported as undeclared on its scorecard.

How it is realised on AWS. Class is a resource attribute set by the SDK or by the agent from a per-service config in the catalogue. A collector processor applies the class policy — pass, sample at rate, aggregate to a count — and emits per-class reduction counters. Class-level retention is expressed as ClickHouse partitions plus S3 lifecycle rules.

What it buys

• An incident's error logs are never sampled away to make room for health checks
• The reduction ratio is visible per service, so nobody has to guess what is missing
• The storm response has a defensible order rather than a random one

What it costs

• Classification is real work at onboarding and is the step most likely to be skipped
• If most services accept defaults, the shed order degenerates towards shedding at random
• Class is another attribute that must be correct, and a mis-classified stream is worse than an unclassified one

Choose differently when. If log volume were small enough to retain in full, class would remain useful for retention and access control but would stop being a reduction mechanism.

Why it holds up over time. The declaration outlives the pipeline: whatever performs the reduction in five years still needs to be told which lines are evidence, and only the emitting team knows.

Lesson. Reduction policy belongs to whoever knows what the data is for; the platform's job is to make the declaration cheap and its absence visible.

ADR-10 · The shed order is declared in advance, and the metric path is not in it

Status: Accepted · Shown on views: 18, 12

When ingest exceeds what the platform can absorb, what is dropped?

Context. The platform's load is positively correlated with the failures it exists to explain: an incident produces an error-log storm, retry amplification and a dashboard stampede at the same moment. Something has to give, and deciding what under pressure means deciding badly. Absorbing everything means sizing ingest and storage for the worst incident rather than the steady state, which is headroom that is idle almost always.

Decision. The platform absorbs 5× steady state for 10 minutes without loss. Beyond that it sheds in a fixed, published order: debug log classes, then access log classes, then trace sampling rate. The metric path is not in the order at all, because shedding it would falsify every count during exactly the incident the counts are for. Every shed volume is counted, attributed to a team, and surfaced in the same interface where the data is missing.

How it is realised on AWS. Gateway processors implement priority admission keyed on signal and class, driven by buffer produce latency and spool depth. Shed counters are per team, per class and per minute, and a shed event writes a gap marker that query results carry through as reduced coverage.

What it buys

• Counts, rates and alerting stay true through the storm
• The loss is bounded, named and attributed rather than arbitrary
• Recovery is a replay from the retained buffer offset, not a permanent hole

What it costs

• The evidence corpus is thinnest exactly when it is most wanted
• Headroom for 5× is real money spent on capacity that is idle almost always
• The order is only as good as the class declarations behind it

Choose differently when. If the buffer could be made deep and cheap enough to absorb any plausible storm — and the resulting freshness lag were acceptable — shedding could be replaced by queueing with a published lag.

Why it holds up over time. The correlation between incident and telemetry volume is structural. Any future platform will face the same choice, and having the order written down before the incident is the transferable part.

Lesson. Decide what you will drop while nothing is on fire, publish it, and exempt the data your decisions are made from.

Storage and retention

The replay point, the substrate, the tiers, and the one class that outlives the rest.

ADR-11 · One durable buffer between admission and every writer is the platform's single replay point

Status: Accepted · Shown on views: 02, 12

What makes a storage failure a recoverable event rather than a permanent hole?

Context. Three stores with different technologies, different failure modes and different maintenance windows are each capable of being unavailable for an hour. If each writer consumes directly from the gateway, an hour of downtime is an hour of lost telemetry, and every store needs its own durability story. A shared buffer converts all of those into lag rather than loss, at the price of one more distributed system in the critical path and a retention window that is real money.

Decision. A single durable, partitioned stream sits between the gateway and every storage writer. It is the only component whose loss beyond its replication is a data-loss event, and everything downstream is replayable from it within its retention. Retention is 72 hours, which is the platform's actual RPO story: a writer can be down for most of a long weekend and lose nothing.

How it is realised on AWS. Amazon MSK across three availability zones, topics per signal, partitioned on service identifier so one service's traffic does not concentrate on one partition, with consumer groups per store. Consumer lag is a first-class alerting signal (view 21), and produce latency drives the shed decision in ADR-10.

What it buys

• Storage maintenance, migration and failure are lag events rather than loss events
• Only one component in the platform carries a real RPO
• A new store or a reprocessing job can be built by replaying history rather than by waiting for new data

What it costs

• A cluster in the critical path of every signal, whose unavailability sheds telemetry
• 72 hours of retention at this volume is a significant standing cost
• Shortening retention to save money quietly shortens every store's recoverability

Choose differently when. If every store had strong native durability and generous internal buffering, the shared buffer would be redundant machinery in the hot path.

Why it holds up over time. The log-as-replay-point pattern has survived every generation of streaming technology because the property it provides — recoverability decoupled from consumer availability — is what makes multi-store pipelines operable at all.

Lesson. Put durability in one place, make everything downstream rebuildable from it, and be explicit that its retention window is the real recovery objective.

ADR-12 · Object storage is the retention substrate; query capacity scales without moving data

Status: Accepted · Shown on views: 12, 14

What does the platform's data actually live on, and what happens when query demand doubles?

Context. Cluster-local storage ties retention to cluster size: keeping data longer means running more nodes, and answering more queries means moving data onto them. At the volumes here that coupling makes every capacity decision a data-migration project, and it makes a query surge during an incident unanswerable by adding capacity — which is precisely when it is needed. The counter-argument is real: querying object storage is slower and needs a caching tier to be interactive at all.

Decision. All long-lived telemetry lives in object storage, with stateless or cache-backed query engines reading from it. Retention becomes a storage-class and lifecycle decision rather than a cluster-sizing one, and storage, query and ingest scale independently of each other.

How it is realised on AWS. Mimir blocks, Tempo blocks and ClickHouse parts all on S3, with lifecycle transitions to Glacier Instant Retrieval for the cold log tier. Query engines run on EKS node groups and EC2 with local NVMe used as cache rather than as the record. Athena serves rehydrated cold queries.

What it buys

• Retention is changed with a lifecycle rule rather than a migration
• A query surge is answered by adding query capacity, with no data movement
• Losing a query node loses cache, never data

What it costs

• Interactive latency depends on a cache that must be warm, which is worst right after a deploy or a failover
• Object request costs are a real line item at this query volume and must be watched
• Three engines each with their own block format on one substrate is three tuning problems, not one

Choose differently when. For a small, latency-critical estate with modest retention, local storage is simpler, faster and cheaper, and the coupling never becomes painful.

Why it holds up over time. Separation of storage from compute has been the direction of travel for every analytical system for a decade, and object storage economics keep improving relative to attached disk.

Lesson. If retention and query capacity are the same knob, you will eventually choose between remembering and answering.

ADR-13 · Cold data is kept raw and rehydrated, rather than downsampled and discarded

Status: Accepted · Shown on views: 14

What happens to telemetry older than the warm tier: is it summarised, or kept and made slow?

Context. Downsample-and-discard is cheap, bounded and permanently forecloses any question about a specific request from six months ago. Keep-and-rehydrate preserves those questions at a storage cost that grows forever and a retrieval latency that makes the data useless in an incident. The honest question is how often anyone genuinely needs raw data older than thirty days, and the honest answer is: rarely, and unpredictably, and usually because of an audit or a regression that was introduced long before it was noticed.

Decision. Metrics are downsampled through declared resolutions — 10 s, 1 min, 5 min, 1 hour — and are never discarded within their retention, with percentile-bearing metrics stored as mergeable sketches so re-aggregation stays correct. Logs are kept raw to 400 days in a cold class and rehydrated on request within 15 minutes. The cold tier is explicitly not an incident tool, and the platform says so rather than letting anyone discover it at 03:14.

How it is realised on AWS. Mimir compactor produces the downsampled resolutions; the response states the resolution served. S3 lifecycle moves log partitions to Glacier Instant Retrieval at 30 days; a rehydration request restores the partitions and exposes them to Athena, with the request queued and its completion notified.

What it buys

• An audit or a long-latent regression is answerable from primary records
• Metric history stays honest across tier boundaries because sketches re-aggregate correctly
• The compliance class already required raw retention, so the marginal decision is smaller than it looks

What it costs

• Cold storage grows monotonically and needs an explicit review cadence
• Fifteen minutes is useless during an incident, and people will discover that at the worst moment
• Rehydration has its own cost that is easy to trigger accidentally

Choose differently when. If two years of usage showed cold rehydration being used a handful of times, downsample-and-discard becomes the right answer and the saving is substantial. This is the ADR most likely to be revisited with data.

Why it holds up over time. Archive storage keeps getting cheaper faster than the questions get easier to anticipate, which tends to strengthen the case for keeping raw over time.

Lesson. You cannot know today which question you will need to answer in a year — but you can know how long you are willing to wait for the answer.

ADR-14 · The compliance class is separable, with its own retention, its own audience and keys held outside the platform

Status: Accepted · Shown on views: 12, 23

Some telemetry must be kept for seven years and read by very few people. Does that obligation apply to everything?

Context. If one retention obligation is applied to the whole corpus, the platform keeps 6 TB a day for seven years to satisfy a requirement that covers perhaps 1% of it. If it is applied by ad-hoc query at deletion time, the obligation is enforced by a script nobody reviews. And subject erasure — deleting a named person's records across every tier including cold — is either designed for or effectively impossible.

Decision. A security- and compliance-relevant log class is declared at emission, stored separately, retained for seven years under storage-level locked retention, and readable by a narrower audience than general telemetry. Everything else expires on its own schedule. Encryption keys are held outside the platform so that crypto-shredding a class is available as a deletion mechanism of last resort, and targeted subject erasure across all tiers completes within 30 days.

How it is realised on AWS. The class routes to its own S3 prefix under Object Lock in compliance mode, with a KMS key the platform's own roles cannot schedule for deletion. Read access is a separate IAM role granted to a small group through IAM Identity Center; every read is written to the query audit first (ADR-16). Erasure is a per-subject key derivation plus tombstone, run as a scheduled job across warm and cold prefixes.

What it buys

• The expensive obligation covers about 1% of the corpus instead of all of it
• Immutability is enforced by the storage layer, not by the code that writes to it
• Erasure has a designed path rather than an escalation

What it costs

• Class declaration is load-bearing for a legal obligation, and a mis-declared stream is a compliance gap
• A narrower audience means real friction for engineers who occasionally need it
• Locked retention means an over-broad class cannot be cheaply undone

Choose differently when. If the whole estate fell under a uniform retention obligation, separation buys nothing and a single locked class is simpler.

Why it holds up over time. Regulations change their durations and their subjects, not their shape: something must be immutable, something must be erasable, and the two must be separable. A design that can express all three survives the rewrites.

Lesson. Scope an expensive obligation to the data it actually covers, and enforce it below the layer that would otherwise be asked to break it.

Query and correlation

How three very different stores answer one investigation, and what every answer must admit about itself.

ADR-15 · Three signal-specific stores behind one query surface, rather than one engine over one store

Status: Accepted · Shown on views: 10, 17

Should the three signals share a physical store and a query language, or be federated behind a common surface?

Context. The three signals have genuinely different access patterns: dense recent time series read by range, sparse text and attribute search over large partitions, and trace assembly by identifier. One columnar substrate can serve all three and will suit none of them well, particularly the metric path, where the query rate and latency budget are most demanding. Federation lets each store be right for its signal and pushes the correlation problem into an index and a query planner that must stitch results with different latencies and different completeness.

Decision. Each signal keeps a store suited to its access pattern. One query surface sits in front of them, providing a common entry point, per-tenant admission, result caching and the completeness labelling in ADR-16. Cross-signal correlation is served by the exemplar index and by trace identifier joins rather than by a distributed join across stores.

How it is realised on AWS. Mimir for metrics, Tempo for traces, ClickHouse for logs and high-cardinality group-by, DynamoDB for exemplars, all reachable through a query frontend that fronts Amazon Managed Grafana. Each store has its own per-tenant limits; the frontend owns admission, caching and the coverage statement on every result.

What it buys

• Each store is tuned for one access pattern rather than compromised for three
• A failure or upgrade in one store degrades one signal rather than all of them
• Per-tenant limits can differ per signal, which matches how the cost actually differs

What it costs

• Three storage systems to operate, upgrade, capacity-plan and be on call for
• Correlation is a designed feature rather than a free consequence of colocation
• A federated result must state coverage honestly when one member returns partial, which is the hardest part of ADR-16

Choose differently when. If one engine could meet the metric path's query rate and latency budget while retaining raw attributes, the single-store design wins on simplicity and on correlation quality together.

Why it holds up over time. The access patterns are properties of the signals and will not converge. What may change is whether one engine can serve all three well — which is the condition to re-test, not the decision to defend.

Lesson. Federate when the access patterns genuinely differ, and then spend the saving on making the crossing between them excellent.

ADR-16 · Every result states its completeness, and a partial answer beats a timeout

Status: Accepted · Shown on views: 17, 21

What does the platform return when it cannot fully answer, and how does the reader know what they are looking at?

Context. Three things routinely make a result less than complete: the data was sampled, a tier or shard did not answer, or the query exceeded its budget. In every case the default behaviour of most systems is to return what they have and say nothing, or to fail with an error. Both are worse than they look. An engineer who computes an error count from a 6% corpus and is not told is wrong by a factor of sixteen; an engineer whose group-by times out learns to stop asking.

Decision. Every result carries its completeness: the sampling rate applied, the tiers consulted, any shard that did not answer, and the resolution of the data returned. A query that exceeds its budget returns a partial result with a stated coverage rather than timing out. Reading production telemetry is an audited action, written before the query executes.

How it is realised on AWS. The query frontend annotates every response with a coverage envelope built from per-store partial-result reporting and the effective sampling rate on the records read; Grafana panels render it as a coverage badge. The audit entry is written to Aurora and copied to the audit store before execution, so an interrupted query is still recorded as an attempt.

What it buys

• An engineer can tell the difference between "there were no errors" and "we did not keep the evidence"
• Exploratory queries stay usable under load instead of timing out
• Access to production telemetry is accountable

What it costs

• Every store must report partial results honestly, which not all of them do natively
• Coverage badges are one more thing on a dashboard that people can learn to ignore
• Auditing before execution adds a write to the read path

Choose differently when. If everything were retained unsampled and queries always completed, coverage would collapse to a shard-availability flag and could be an exception rather than a field.

Why it holds up over time. A result that cannot describe its own limits is a hazard in any data system, and the pressure to state provenance and completeness is increasing everywhere, not decreasing.

Lesson. The most dangerous answer is a plausible one with an unstated caveat. Make the caveat part of the answer.

Alerting and evaluation

The plane that pages a human, what it is allowed to read, and the failure that looks like health.

ADR-17 · Alert evaluation runs on its own capacity and reads the store directly

Status: Accepted · Shown on views: 10, 19, 25

What must keep working when the dashboards do not?

Context. The console and the alert evaluator both read the metric store, and the easy design has both go through the query frontend so that caching, admission and authorisation are implemented once. It also means that a dashboard stampede during an incident, or a bad deploy of the frontend, can stop alerts from firing — a failure that is invisible until a page that should have arrived did not. Alerting is the one function whose absence cannot be noticed by its users.

Decision. The alert evaluation plane runs on its own node groups, with its own scaling and its own deployment cadence, and reads the metric store directly rather than through the query frontend. Interactive query carries 99.9% monthly and alert evaluation carries 99.99%. On a control-plane outage the evaluator continues on last-known rules and never fails an evaluation into silence; every missed evaluation is recorded.

How it is realised on AWS. Mimir ruler on a dedicated EKS node group with no shared capacity with the queriers, its rules loaded from the control plane and cached locally. Missed evaluations, rule-group evaluation duration and the ruler's own liveness are reported to the self-telemetry account in view 19.

What it buys

• Paging survives a query-tier outage, a dashboard stampede and a console deploy
• Alerting and query can be scaled and released on different schedules
• The availability targets in the requirement are achievable rather than aspirational

What it costs

• Dedicated capacity that is idle most of the time and is the first thing a cost review proposes merging back
• Caching and authorisation are implemented twice
• Two code paths onto the same store means two places a query-semantics change has to land

Choose differently when. If the query tier could be made genuinely isolated per workload class — hard multi-tenancy with guaranteed capacity per class — one tier with a reserved class would give the same guarantee more cheaply.

Why it holds up over time. The principle that the mechanism which raises the alarm must not share fate with the mechanism people are using is older than any of this software, and it is the first thing lost when a platform is optimised.

Lesson. Whatever must work during the incident should not share capacity with whatever everyone uses during the incident.

ADR-18 · Absence of signal is a first-class alert condition, driven by expected emission

Status: Accepted · Shown on views: 21, 25

A service stops emitting. What tells anyone?

Context. To a threshold rule, a stopped exporter and a perfectly healthy service look identical: no data above the threshold. This is the failure class that looks like health, and it is the one that survives longest in production, because nobody investigates a quiet dashboard. It is also the failure class an observability platform is uniquely obliged to catch, since nobody else can.

Decision. The platform tracks expected emission per producer, derived from the service catalogue and from observed history, and alerts on absence as a condition distinct from any threshold. The same mechanism covers the platform's own silence through the dead-man switch in ADR-22. Absence alerts are owned by the producing team and are not suppressible without an expiring, recorded justification.

How it is realised on AWS. A per-service emission expectation is materialised from the catalogue and from a rolling observed baseline; a rule group evaluates absence over a window scaled to each service's normal cadence. A newly onboarded service gets its expectation from the catalogue before it has any history.

What it buys

• The failure that looks like health has a detector
• A broken instrumentation deploy is caught in minutes rather than at the next incident
• The platform's own silence is covered by the same idea

What it costs

• Services with genuinely bursty emission generate false absence alerts and are the hardest to baseline
• This is the alert most likely to be tuned down for noise, which would remove the control entirely
• A catalogue entry that is wrong produces an expectation that is wrong

Choose differently when. If every service emitted a synthetic heartbeat at a fixed cadence as a platform convention, absence detection becomes trivial and the baseline is unnecessary. That is a cheaper design and requires estate-wide discipline this one does not assume.

Why it holds up over time. Detecting silence is a permanent requirement of monitoring, and the techniques improve while the obligation does not change.

Lesson. Every monitoring system alerts on things going up. The one that matters alerts on something that should be there and is not.

Economics and governance

Who pays, who decides, and what the platform is allowed to refuse.

ADR-19 · Chargeback with soft quotas, and a hard quota only on cardinality

Status: Accepted · Shown on views: 06, 22

Is telemetry cost enforced as a limit, or attributed as a bill?

Context. A hard quota bounds the platform's spend absolutely and makes the platform the thing that dropped a team's data during their worst hour. A chargeback keeps all the data and turns cost into a conversation weeks later, by which time the labels are in production and the money is spent. The two failure modes are genuinely different in kind: one loses evidence, the other loses money. Cardinality is the exception, because an unbounded label is not merely expensive — it degrades the store for every other tenant.

Decision. Ingest volume, retention and query are attributed daily to owning teams with soft quotas that raise alerts and appear on scorecards, not rejections. Cardinality carries a hard quota at the gateway, because it is the one dimension where one team's excess is another team's outage. Incident-time raises to sampling rate and budgets are self-service, bounded by a per-team ceiling and an automatic expiry.

How it is realised on AWS. The usage meter writes bytes, active series, retained spans and query cost per team per day into the cost ledger, reconciled against the AWS Cost and Usage Report through Athena. Soft-quota breaches open a scorecard item; cardinality breaches are the enforcement events of ADR-06. Incident raises are an API call recorded with an actor and an expiry.

What it buys

• An incident is never made worse by a budget control
• The shared metric store is protected from a single team's mistake
• Teams see the cost of a change at the moment they propose it

What it costs

• Soft quotas only work if the organisation acts on attributed cost, which is a cultural bet
• The platform's total spend is not bounded by construction, only steered
• Incident raises are a real budget leak if the expiry is not enforced

Choose differently when. If attributed cost were repeatedly ignored and spend kept growing, hard quotas with a fast incident-time raise path become the necessary answer. This is the decision most likely to be revisited by circumstance rather than by argument.

Why it holds up over time. The tension between bounding cost and preserving evidence is permanent. Naming which dimension is a shared-resource problem and which is only a money problem is the part that transfers.

Lesson. Enforce hard limits only where one tenant's excess is another tenant's outage; everywhere else, send the bill.

ADR-20 · Query cost is charged to the reader, and unread data is the headline waste class

Status: Accepted · Shown on views: 06, 21

Who pays for an expensive dashboard, and which reduction can actually be argued for?

Context. Producing teams have no control over who queries their data or how expensively. If query cost lands on the producer, the incentive is to emit less so that others cannot ask — exactly backwards. Meanwhile the only reduction that can be argued without guessing is the one supported by evidence, and the available evidence is what nobody reads: a service whose telemetry has not been queried in ninety days is paying for storage that has answered no question.

Decision. Query cost is attributed to the querying team, not the producing team. Last-queried statistics are maintained per service and per signal, and unread data is reported as the platform's primary waste class. Reduction proposals lead with unread data before they touch sampling rates or retention on data that is actually used.

How it is realised on AWS. The query frontend meters bytes scanned, object requests and compute per query and attributes them to the caller's team. Last-queried timestamps are maintained per service, signal and tier and surfaced on the monthly scorecard alongside cost.

What it buys

• An expensive dashboard has a named owner
• Cost reduction has an argument behind it rather than an across-the-board percentage
• Producing teams are not penalised for being useful to others

What it costs

• Charging for queries can discourage exactly the exploration the platform exists for, and needs a generous free allowance
• Last-queried is a weak signal for data kept against an audit obligation, which is read approximately never and must be kept
• Metering every query adds work to the read path

Choose differently when. If investigation were being suppressed by query cost, the right move is to absorb query cost centrally and keep only the attribution for visibility.

Why it holds up over time. Whoever causes a cost should see it. That principle does not age, even as the specific meters change.

Lesson. Attribute cost to the party whose behaviour you want to change, and reduce first where you have evidence rather than where it is easiest.

ADR-21 · The service catalogue is the front door, and an unclaimed producer is quarantined

Status: Accepted · Shown on views: 05, 10

Where do ownership, budget, retention class and default dashboards come from?

Context. Every per-team mechanism in this architecture — budgets, chargeback, absence detection, access control, alert routing — needs to know who owns a service. Maintaining that inside the observability platform creates a second registry that diverges from the first within a quarter. Not maintaining it at all means onboarding becomes a platform-team ticket, which does not scale to 900 services and makes the journey in view 05 impossible.

Decision. The service catalogue is authoritative for ownership, environment, retention class and budget, and the platform syncs from it rather than holding its own copy. Onboarding is catalogue-driven: identity, defaults, dashboards and budgets are applied without a platform-team ticket. Telemetry from a producer with no catalogue entry is accepted into a low-budget quarantine tenant rather than into the general pool, so it is neither lost nor able to consume shared capacity.

How it is realised on AWS. A sync job reconciles the catalogue into Aurora on a schedule and on change events. Default dashboards and SLO scaffolding are generated per service from the catalogue entry. The quarantine tenant has a small fixed budget, short retention, and appears on an unclaimed-producers report.

What it buys

• A new service is observable the day it ships with no platform-team involvement
• Every per-team mechanism has one consistent source for who owns what
• Unmanaged producers are visible and bounded instead of invisible or rejected

What it costs

• The platform inherits the catalogue's data quality problems
• A catalogue outage degrades onboarding and absence expectations, though not ingest or query
• The quarantine tenant is a place things can sit forever unless the unclaimed report is acted on

Choose differently when. Without a credible catalogue, the platform would have to own a minimal ownership registry of its own — and should then treat it as a temporary liability with a plan to retire it.

Why it holds up over time. Service catalogues change shape and vendor, but the rule that the platform reads ownership rather than asserting it keeps the coupling one-directional and replaceable.

Lesson. Consume the organisation's source of truth rather than forking it, and give the records that do not fit a bounded home instead of the bin.

Operations and assurance

Running a platform that depends on the estate it observes, and releasing to 12,000 hosts nobody asked.

ADR-22 · Self-telemetry runs in a separate account and region, on managed services

Status: Accepted · Shown on views: 19, 21

The platform runs on the estate it observes. Who watches it?

Context. Self-observation is circular by construction: the platform's own metrics flow through its own pipeline into its own store and are alerted on by its own evaluator. Every one of those steps fails during the incidents that matter most, and the platform then reports itself healthy because nothing is arriving to say otherwise. Running a second copy of the same stack does not help if it shares an account, a region, an EKS control plane or a service limit.

Decision. A minimal self-telemetry and alerting path runs in a separate AWS account and region on managed services, sufficient to alert that the platform itself is blind. A heartbeat canary writes a probe through the full ingest path and reads it back through the full query path; a dead-man switch pages the platform team on silence. The stack shares no EKS control plane, no account limit and no scaling event with the platform it watches.

How it is realised on AWS. Amazon Managed Service for Prometheus and Amazon Managed Grafana in a separate account and region, receiving a small set of platform health, freshness and completeness series plus the canary's round-trip result. The dead-man switch alerts the paging platform directly when the heartbeat stops.

What it buys

• The platform's blindness is detectable from somewhere that cannot go blind with it
• The round-trip canary tests the whole path, not the components
• The alert reaches the paging platform without traversing anything that is failing

What it costs

• A second account, a second bill and a second stack to maintain for something that does almost nothing on a good day
• It is the most likely line item to be cut in a cost review, which would silently remove the control
• Deliberately low fidelity: it can say the platform is blind and very little about why

Choose differently when. If the paging platform itself offered a heartbeat-absence check, much of this could collapse into that, and the separate stack would only be needed for the round-trip probe.

Why it holds up over time. The circularity is structural for any observability platform on the infrastructure it observes. Whatever the technology, the answer is always somewhere with a different fate.

Lesson. A system cannot be the only witness to its own failure. Buy the smallest possible independent witness and protect it from the cost review.

ADR-23 · Config, rules and budgets are code with gates; agent rollouts ride rings

Status: Accepted · Shown on views: 20

How does a change to a collector pipeline, an alert rule or a budget reach production safely?

Context. The platform's configuration is more dangerous than its code. A collector pipeline change can silently stop a signal; a budget change can start rejecting a team's metrics; an agent release touches 12,000 hosts that no service team asked to have touched. None of these look like deployments in the usual sense, and all of them are.

Decision. Collector configuration, alert and SLO rules, budgets and retention classes live in Git and are released through a pipeline with gates: rules must read the complete path (ADR-03), a cardinality estimate runs before a metric reaches production, config is schema-validated and replayed against a recorded corpus, and a cost delta is previewed on the pull request. Environments progress staging, canary region, then waves. Agent releases roll out in rings of 1%, 10% and 100% of hosts, and config rollback is a single action.

How it is realised on AWS. Git as the source of truth with Argo CD applying per-region manifests. Rule unit tests run through the same evaluator as the ruler. Replay runs a recorded telemetry corpus through the candidate pipeline and diffs the output. Agent rings are node-group labels driving the DaemonSet update.

What it buys

• The completeness boundary and the cardinality budget are enforced by the release process
• Every change is reviewed, tested against a corpus, and revertible
• An agent regression is caught at 1% of hosts

What it costs

• Replay only catches what the corpus contains; an unrecorded signal shape passes every gate
• Config rollback is one action, but rolling an agent back across 12,000 hosts is not
• Gates add latency to urgent changes, which is a real cost during an incident

Choose differently when. In a small estate with a handful of collectors, console editing with an audit trail is proportionate and much faster.

Why it holds up over time. Configuration-as-code has outlived the tools that popularised it because the property it provides — review, test, revert — is what makes any change survivable.

Lesson. Configuration that can stop a signal or start a page deserves the same pipeline as the code that emits it.

ADR-24 · Every plane fails static on control-plane loss, and never fails silent

Status: Accepted · Shown on views: 10, 25

What happens to collection, evaluation and query when the control plane is unavailable?

Context. The control plane holds rules, budgets, retention classes and the catalogue mapping. It is the only non-derived state in the platform, and it is also the least critical to serving: nothing about ingesting a sample or evaluating a rule needs it to be reachable, provided the last-known values are held locally. The dangerous design is one where a control-plane lookup is in the serving path, because then a small store's outage becomes the platform's outage.

Decision. Collection, admission, storage, query and alert evaluation all hold last-known configuration locally and continue operating on it when the control plane is unavailable. Only changes are impossible. Nothing fails closed into silence: an evaluator with stale rules keeps evaluating and reports staleness, and a gateway with stale budgets keeps admitting under the last-known limits. The control plane is therefore allowed the weakest availability target in the platform at 99.5% monthly.

How it is realised on AWS. Every tier caches its configuration and refreshes on an interval, with the cached values marked with their age. Argo CD and Git provide a second source for recovery. Configuration staleness per tier is a reported metric, and a staleness threshold is itself an alert condition.

What it buys

• The control plane can be maintained, migrated or lost without a serving incident
• The availability targets are layered honestly, with the weakest where it matters least
• Staleness is visible rather than assumed

What it costs

• Running on stale budgets and classes can mean enforcing yesterday's policy for hours
• A cache in every tier is a cache in every tier: another thing to invalidate, size and reason about
• A long control-plane outage blocks urgent changes — including incident-time sampling raises

Choose differently when. If policy changes were safety-critical and had to take effect immediately — a privacy control rather than a budget — failing closed on unreachable policy becomes the right choice for that specific class.

Why it holds up over time. Fail-static is the standard answer wherever a control plane governs a data plane, and it becomes more important as the data plane grows relative to the control plane.

Lesson. Keep the store that holds your policy out of the path that serves your traffic, and make the age of what you are running on visible.

Every package used, in one table

Every product and package named on the views, what it is, the job it does in this design, and what was considered instead.