Architecture Views

If someone asks for last quarter's availability, can the platform compute it from what it kept?

Context

The natural thing to store is the answer: a availability percentage per minute, per hour, per day. It is what people want to read, it is what every dashboard displays, and it is one number instead of two. It is also unusable. A ratio cannot be re-aggregated: the mean of twelve hourly availability figures is not the day's availability unless every hour carried identical traffic, and traffic is never identical. Worse, a stored ratio has thrown away the denominator, so there is no way to express a budget in the only unit that is actually spendable — failed requests. Every later capability in this design depends on being able to re-reduce the same underlying counts over a different window, a different exclusion overlay or a different predicate, and all of them become impossible rather than merely expensive the moment the division happens before storage.

Decision

The unit of storage is a pair of integers per SLO per minute: good_count and valid_count, plus a coverage flag. Ratios are computed at read time, never persisted. Every window figure — rolling, calendar, attributed interval, excluded or unexcluded — is a sum of the same two columns over a different set of rows.

How it is realised on AWS

The sli_bucket table in Azure Data Explorer is keyed on (slo_id, minute_utc) with good_count, valid_count, coverage and source_class. Window reduction is a Kusto summarise over a time range; the budget is total minus consumed where both are counts. The budget projection in Redis holds the summed counters and the derived percentage together, with the counters authoritative — a reader that wants to recombine windows reads the counters, and a reader that wants to display a figure reads the percentage.

Options weighed

• ChosenStore good and valid counts separately: Two integers per bucket instead of one float; everything downstream stays recomputable and the budget is expressible in failed requests
• RejectedStore the ratio per bucket: Halves the storage and makes correct re-aggregation across windows impossible, which silently removes rolling windows, exclusion overlays and interval attribution
• RejectedStore raw request events and aggregate at query time: Maximum flexibility at two orders of magnitude more storage and ingest; the platform would then own collection, contradicting ADR-02
• RejectedStore counters plus a pre-computed ratio as the authority: Invites readers to trust the ratio and quietly reintroduces the re-aggregation bug the counters were kept to avoid

Consequences

What it buys

• Any window, any exclusion overlay and any corrected predicate can be recomputed from retained data
• The budget is expressible as a count of permitted failures, which is what makes it feel spendable rather than abstract
• Interval attribution is a sort over the same two columns rather than a separate pipeline

What it costs

• Two columns to keep consistent, and a reader who sums ratios by hand will still get a wrong answer the platform cannot prevent
• The percentage everyone wants to read is computed on every query, so the read path carries arithmetic a stored ratio would have avoided

Choose differently when

If the platform only ever had to answer "what is the availability of this window, right now" — no rolling windows, no exclusions, no retroactive correction, no interval attribution — a stored ratio would be adequate and cheaper. In practice the first request after launch is always "what happened on Saturday", which needs the counts.

Why it holds up over time

This is arithmetic, not engineering. Any future store can re-aggregate counters and none can un-divide a ratio, so this decision should outlive every technology named in the package. It is also the decision most often skipped, because the ratio is the thing people want to see and the cost of storing it looks like a saving.

When the reliability authority and the measurement pipeline fail together, what is left?

Context

The commercially obvious design has the SLO platform own collection: agents in the fleet, its own time-series store, its own query language, SLOs computed over data it controls end to end. It is what several products do, it removes an integration, and it tightens the latency from event to figure. It also makes one system simultaneously the measurement path and the reliability authority, which has two consequences. Operationally, its outage is both an observability outage and a loss of the signal that would tell you an observability outage is happening. Epistemically, it can no longer distinguish "I did not receive data" from "there was no bad traffic", because it is the thing that would have received it — and a platform that cannot represent its own blindness will report blindness as health.

Decision

Collection belongs to the observability platform. This platform's inputs are pre-aggregated counters arriving on a stream, and its outputs are derived figures and signed verdicts. It retains aggregates and derivations; it retains no raw signal, no traces and no logs of the measured systems. Enforcement, symmetrically, belongs to the deployment system (ADR-12).

How it is realised on AWS

Azure Monitor and Azure Monitor managed service for Prometheus remain the system of record for telemetry. Recording rules in the measurement plane emit the outcome cube to Event Hubs; the platform's bucketer consumes it. Nothing in the platform scrapes, and no agent is deployed to a measured workload. The layered view draws the measurement plane as the bottom band, labelled "outside", so no reader of the set mistakes it for a component.

Options weighed

• ChosenConsume aggregates; own no collection: One integration and a dependency on someone else's uptime, in exchange for reproducibility and the ability to describe blindness
• RejectedOwn collection end to end: Tighter latency and no integration; makes the reliability authority a hard dependency of the measurement path and unable to represent its own blindness
• RejectedConsume aggregates but keep a parallel collection path for critical journeys: Two sources of truth for the same SLO, and the first disagreement between them is unresolvable
• RejectedBuy a managed SLO product that owns both collection and enforcement: Fastest to a figure; concentrates measurement, derivation and enforcement in one vendor outage

Consequences

What it buys

• A measurement outage is describable as a coverage deficit rather than invisible as a good month
• Every published figure is a reproducible function of retained aggregates plus a versioned definition
• The platform stays small enough that it is not the most operationally demanding thing in the estate

What it costs

• The platform cannot fix bad telemetry; it can only refuse to launder it, which moves some problems to a team that does not own the budget
• The outcome cube's dimensions are chosen in a system this platform does not control (ADR-03)
• Event-to-figure latency carries an extra hop, which is why budget freshness is specified at 60 s rather than seconds

Choose differently when

If blind minutes became routine rather than exceptional — coverage regularly below the floor across many SLOs — the boundary would stop paying for itself, because a platform that answers insufficient-data most of the time is not an authority. At that point owning collection for the critical journeys would be the correct response, and the decision should be formally revisited rather than quietly eroded.

Why it holds up over time

"Compute and publish, do not measure and do not enforce" is a statement about which systems may depend on which, not about any product. It is also the single most useful test to apply to future proposals: a vendor platform that owns collection and gating contradicts it, and that contradiction is the thing to argue about rather than the feature list.

When a team wants to change what "good" means, how much history can they take with them?

Context

Having decided to consume aggregates (ADR-02), the obvious contract is the one the platform needs: each source emits good_count and valid_count per minute per SLO. It is minimal and cheap. It also freezes the good-event predicate at the point of emission, which means the first time a team wants to tighten a latency threshold from 800 ms to 500 ms, there is no history to recompute against — the distinction was discarded upstream before the platform ever saw it. Request-level events would preserve everything, at two orders of magnitude more ingest and storage, and would drag the platform back across the collection boundary. The middle position is to notice that almost every predicate change teams actually want moves along one of two axes: which status classes count as failures, and which latency threshold counts as slow.

Decision

Sources emit a small declared cube per SLO per minute — counts bucketed by status class and by latency bucket — rather than a single good/valid pair. The good-event predicate becomes a projection over the cube's declared dimensions, so a predicate change along those dimensions is recomputable over retained history at pre-aggregation cost. A change outside the declared dimensions is not recomputable, and the platform says so rather than pretending.

How it is realised on AWS

Prometheus recording rules in the measurement plane emit a histogram-shaped record: for each SLO and minute, counts keyed by status class (2xx, 4xx, 5xx, client-cancelled) and by latency bucket on a fixed ladder. The bucketer flattens the cube into sli_bucket rows keyed by (slo_id, minute_utc) with the cube retained as a nested column, so window reduction sums the cells the current predicate selects. Cardinality is bounded by the admission check in ADR-15: the cube's cell count per SLO per minute is capped, which is what keeps this affordable at 1.73 million buckets a day.

Options weighed

• ChosenA small declared outcome cube: A few cells per bucket instead of two integers; keeps retroactive redefinition possible along the axes teams actually use
• RejectedA single good/valid pair: Cheapest contract; makes every predicate change forward-only, which is the complaint that arrives in month two
• RejectedRequest-level events into the platform: Any predicate, any time, full attributability; two orders of magnitude more cost and it crosses the collection boundary (ADR-02)
• RejectedCube for critical journeys, pair for the long tail: Tempting on cost; produces two classes of SLO with different capabilities and a migration whenever a journey is promoted

Consequences

What it buys

• Tightening a latency threshold or reclassifying a status code recomputes over 13 months of history with no upstream change
• Cost stays within a small multiple of the minimal contract, because the cube is bounded rather than open
• The limits of recomputability are explicit — a team can be told which changes are free and which need a source change

What it costs

• The cube's dimensions are chosen once, upstream, by a team that does not own this platform, and a dimension nobody thought to emit is unrecoverable
• A nested column per bucket is more complex to reduce than two integers, and the reduction has to stay deterministic for ADR-01 to hold
• Teams will occasionally want a predicate the cube cannot express, and the answer is a genuine upstream change rather than a configuration one

Choose differently when

If storage and ingest for request-level events became cheap enough that 2.5 million events a second could be retained for 13 months without a separate budget conversation, the cube would be an unnecessary compromise and event-level ingest would dominate it — though the collection-boundary objection in ADR-02 would still stand, so the events would have to arrive from the measurement plane rather than be collected here.

Why it holds up over time

The cube is a bet about which questions get asked, not about a technology. The specific dimensions will be revised; the principle — pre-aggregate along the axes along which the definition is likely to move, and declare the limits — transfers to any pre-aggregation decision in any system.

What does the platform publish when it did not receive the data?

Context

A minute with zero valid events is ambiguous: it may be three in the morning on a low-traffic journey, or the metrics pipeline may have stopped. The convenient implementations are both wrong. Treating the minute as good — a ratio of 0/0 rendered as 100% — means a stopped scrape presents as a perfect month, which is the single most common way this class of platform lies. Treating it as bad means every quiet night burns budget and the SLO becomes unusable for anything but the busiest journeys. Interpolating from neighbouring minutes is worse than either, because it manufactures evidence. The deeper point is that a reliability authority has an asymmetric cost function: a wrong figure is materially worse than no figure, because a wrong figure is acted on.

Decision

A bucket with no valid events is recorded in a third state, no-data, which is neither good nor bad and is excluded from both numerator and denominator. Coverage — the fraction of expected buckets present — is computed per window and published alongside every figure. Below a declared coverage floor the verdict is the typed value insufficient-data rather than a number, and reliability alerts for that SLO are suppressed in favour of a measurement-failure alert (ADR-10).

How it is realised on AWS

sli_bucket carries a coverage column with values ok and no-data. Window reduction counts expected minutes against present-and-ok minutes to produce coverage_pct, which is written into the budget projection and returned in every verdict and query response. The verdict API evaluates coverage before anything else and short-circuits to insufficient-data below 95%. Low-traffic journeys declare an aggregation rule — a minimum valid-event count per evaluation period — rather than being silently imputed.

Options weighed

• Chosenno-data as a third state, with a published coverage floor: Costs a column, a floor nobody can derive from first principles, and a fourth verdict value; makes blindness representable
• RejectedTreat an empty bucket as good (0/0 = 100%): Simplest and the most dangerous: a stopped pipeline is indistinguishable from a perfect month
• RejectedTreat an empty bucket as bad: Fails safe in principle; makes every low-traffic journey permanently in breach and the platform unusable for the long tail
• RejectedInterpolate from neighbouring minutes: Produces a plausible figure with no evidence behind it, which is the one outcome worse than no figure

Consequences

What it buys

• A metrics outage is visible as itself rather than as good news
• Every figure carries its own confidence, so "11% remaining" is never read as more certain than the data behind it
• Regional recovery has an honest intermediate state: insufficient-data for 90 minutes while the projection rebuilds (ADR-16)

What it costs

• The 95% floor is a judgement with no principled derivation, and SLOs will sit just above and just below it for reasons unrelated to reliability
• insufficient-data is a fourth state every consumer has to handle, and a consumer that treats it as healthy has reintroduced the original bug
• A low-traffic journey needs an explicit aggregation rule, which is extra work at definition time

Choose differently when

If every SLO in the estate were on a journey with continuous high traffic and a measurement pipeline with its own hard availability guarantee, the ambiguity would largely disappear and a two-state bucket would be adequate. That is not true of the long tail of 400 services, and it is never true during an incident — which is exactly when the figure is read.

Why it holds up over time

Telemetry pipelines will keep breaking in new ways indefinitely. A design in which absence is representable degrades gracefully under failures nobody has imagined yet; one in which absence is indistinguishable from success fails silently under all of them. This should be among the last decisions anyone revisits.

What happens when the same minute's data arrives twice, and when it arrives too late?

Context

Aggregates arrive over a stream, which means they arrive at-least-once, out of order, and sometimes long after the minute they describe. Assigning them to buckets by arrival time is simpler and wrong: a backlog drained after a pipeline incident would attribute an hour of old failures to the minute the backlog cleared, which both hides the real interval and manufactures a fake one. Event-time assignment is correct but leaves two questions: how long a bucket stays open for late arrivals, and what stops a replayed batch from double-counting. An unbounded reopen window means no figure is ever final; an exactly-once delivery guarantee is not available from any stream the platform would use.

Decision

Every aggregate is assigned to its bucket by event time. A bucket accepts late arrivals for 10 minutes after its close and is then immutable; data arriving later is recorded as a coverage deficit rather than silently dropped or silently applied. Idempotency is structural: (slo_id, minute_utc) is the bucket's primary key, so a replayed batch is a key collision resolved by overwrite-with-identical-value rather than an addition.

How it is realised on AWS

The bucketer consumes Event Hubs partitions with checkpointing, derives minute_utc from the record's own timestamp, and upserts sli_bucket rows on (slo_id, minute_utc) — the write is the full cell set for that minute from that source, not a delta, which is what makes a replay a no-op. A late record whose bucket has passed the horizon increments the window's coverage deficit. An input batch that would move a closed window's figure beyond a declared tolerance is quarantined with a typed reason rather than applied, and the quarantine is a retained, replayable store rather than a dead-letter queue.

Options weighed

• ChosenEvent time, 10-minute horizon, idempotent upsert on (slo_id, minute): Correct attribution, bounded finality, and double-counting made impossible by the key rather than by the consumer's care
• RejectedArrival-time bucketing: Trivial to implement; a drained backlog invents an incident and hides the real one
• RejectedEvent time with an unbounded reopen window: Never loses a record; no figure is ever final, so nothing can be sealed and the compliance record becomes provisional forever
• RejectedAdditive deltas with a de-duplication table: Standard streaming pattern; replaces a primary key with a second system that has to be correct, and its failure mode is a silently wrong count

Consequences

What it buys

• An interval attribution points at the minutes users actually suffered, which is what makes view 04's trough answerable
• A replay — after a bucketer crash, a source backfill or an operator retry — cannot change a figure
• Windows become final, so snapshots can be sealed and the compliance record means something (ADR-14)

What it costs

• Data later than 10 minutes is lost as measurement and visible only as reduced coverage, which makes the figure slightly pessimistic
• The upsert must carry the full cell set per source per minute, so the write is larger than a delta
• Multiple sources writing the same SLO's minute need source_class in the key path or they overwrite each other

Choose differently when

If the upstream pipeline offered a hard bound on delivery latency — every record within 30 seconds or not at all — the horizon could shrink to near zero and the coverage-deficit path would be mostly dead code. Conversely, if the platform ever had to accept bulk historical imports as a normal operation, the horizon would have to become a per-source declaration rather than a constant.

Why it holds up over time

Event-time semantics and key-based idempotency are the two things stream processing has converged on for good reasons, and neither depends on a product. The horizon's specific length is a tuning parameter that will change; the existence of a declared horizon, after which a figure is final, is what makes everything downstream possible.

When the SLO goes blind, who is woken, and what are they told?

Context

Once blindness is representable (ADR-04), the alerting question follows: what happens when coverage collapses? Doing nothing is unacceptable — a silently blind SLO is an unmonitored journey. Firing the reliability alert is worse than nothing, because it tells the on-call engineer that users are suffering when the truth is that the platform cannot see. That is the single worst minute in the whole system: the engineer is awake, under time pressure, and holding a signal that points at the wrong system entirely. The diagnosis they need — "this is a metrics problem, not a service problem" — is a different conclusion, reached by a different person, with a different runbook.

Decision

Below the declared coverage floor, reliability alerts for that SLO are suppressed and a distinct measurement-failure alert is raised instead, routed to the platform rotation rather than the service rotation. The two classes are never merged, and the measurement-failure alert names the SLO, the coverage figure and the source class that stopped.

How it is realised on AWS

The burn-rate evaluator checks coverage before evaluating any rule and emits one of two alert types. Measurement-failure alerts route through a separate Azure Monitor action group to the platform rotation; reliability alerts route to the owning service's rotation. The suppression is recorded on the alert_decision row with suppressed_reason, so a suppressed window is explicable afterwards and does not silently vanish. Nights attributed to measurement failure are excluded from the page-precision figure in ADR-09, so a broken pipeline cannot degrade the statistics that justify the alerting configuration.

Options weighed

• ChosenTwo alert classes, separate rotations, with suppression recorded: An extra alert type and routing to maintain; the engineer is told which system is broken
• RejectedOne alert class covering both: Simplest routing; puts every reader in the position of being unable to tell an outage from a blind SLO
• RejectedSuppress silently below the floor: No false pages at all; an unmonitored journey nobody is told about, which is the failure this platform exists to prevent
• RejectedDegrade the reliability alert's severity instead of reclassifying: One mechanism, less plumbing; a low-severity reliability alert still points the reader at the wrong system

Consequences

What it buys

• The page names the broken system, so triage starts in the right place
• Pipeline health becomes a platform concern with its own rotation rather than an unexplained service symptom
• Alert precision stays a meaningful figure, because measurement failures do not pollute it

What it costs

• Two rotations must both exist and both be staffed, which is an organisational dependency this design assumes
• An SLO can sit blind and un-alerted to its owning team, who find out from the coverage figure rather than from a page
• A partial measurement failure above the floor still produces a reliability alert computed on thin data

Choose differently when

If one team owned both the measurement pipeline and the services being measured, the split would be bureaucratic rather than useful and a single rotation with a well-worded alert body would be adequate. The split exists because the two systems have different owners.

Why it holds up over time

The distinction between "the thing is broken" and "my view of the thing is broken" is permanent in any monitoring system. Products and routing will change; the requirement that those two conclusions reach different people with different runbooks will not.

When a target turns out to have been defined wrongly for three months, what happens to those three months?

Context

A console where SLOs are edited in place is the fastest thing to build and the easiest to use. It also makes a definition change indistinguishable from a definition correction, destroys the association between a published figure and the text that produced it, and leaves no review step at the one moment when review matters most — because an SLO is a commitment, and widening a denominator is a commitment change dressed as a configuration tweak. The alternative is to treat the definition as code: text in the owning team's repository, reviewed by pull request, reconciled into a registry that is a projection of it rather than a system of authorship.

Decision

SLO definitions are declarative text in the owning team's repository. A reconciler reads merged definitions and writes versioned, immutable records into the registry, each with an effective-from timestamp. Prior versions remain readable forever, so every historical figure stays attributable to the exact text that produced it, and a correction produces a corrected history rather than a silent rewrite.

How it is realised on AWS

A Container Apps job polls the definitions repository and reconciles merged changes into Azure SQL, writing a new slo_version row rather than updating one. slo carries current_version as an FK; every budget_window and verdict records the version_id it was computed under. The admission checks in ADR-15 run in the reconciler, so a definition that cannot be measured fails the merge rather than entering the registry. The console is read-mostly: it can pause an SLO and request an exclusion, but it cannot author a predicate.

Options weighed

• ChosenRepository-authored text, reconciled into a versioned registry: Review at the right moment, permanent attribution, and corrections that are visible as corrections; slower to change than a console
• RejectedConsole editing with an audit log: Fast and comfortable; the audit log records that a change happened without making the prior definition's figures recomputable
• RejectedVersioned records authored through an API, no repository: Keeps immutability and loses the review step, which is most of the value — nobody reviews an API call
• RejectedRepository as the only store, no registry: Purest GitOps; makes every verdict read a repository read, and the registry's transactional guarantees are needed for exclusions and policy

Consequences

What it buys

• A wrong definition is corrected and the history recomputes, instead of the error being frozen into the record
• Widening a denominator is a reviewed change with a named author, which is the only structural counterweight to targets drifting easier
• Every figure in a monthly report can be traced to the text in force at the time

What it costs

• Changing an SLO takes a pull request, which teams will find slow in exactly the moment they want it to be fast
• The registry can lag the repository, so there is a reconciliation window during which the two disagree
• Definition ownership stays with the service team, which makes targets achievable and comparability across 400 services weak

Choose differently when

If SLO definitions needed to change inside minutes — an incident-time adjustment, a dynamically generated SLO per tenant — a repository round trip would be the wrong mechanism and an API with versioned records would dominate. Nothing in this problem has that shape: an SLO that changes hourly is not a commitment.

Why it holds up over time

"The definition is reviewed text and the registry is its projection" survives any change of version-control system or registry technology. The specific reconciler is disposable; the claim that authorship lives outside the serving system is what keeps corrections honest.

Should the cost of Saturday's outage decay gradually, or reset on the first of the month?

Context

The two window shapes produce opposite incentives over identical arithmetic. A trailing 28-day rolling window never resets: an incident's cost decays minute by minute, which matches how trust in a service actually recovers, and means a team can be frozen for four weeks over one bad Saturday. A calendar month resets on a date: it is comprehensible, it aligns with every other report in the business, and it rewards holding risky changes until the first while making the last week of a bad month unshippable. Both can be computed from the same counters (ADR-01) and both can be published. Only one can drive the freeze, because two verdicts for the same SLO is not a verdict.

Decision

Both windows are computed and published for every SLO. The trailing 28-day rolling window is canonical for the verdict and therefore for the freeze; the calendar month is the reporting and compliance window and is the one sealed at close (ADR-14). Every response states which window produced the figure it carries.

How it is realised on AWS

budget_window rows carry kind as rolling or calendar with their own open and close timestamps, both reducing over the same sli_bucket rows. The budget projection materialises both. The verdict API reads the rolling figure and names it explicitly in the response; the monthly report and the sealed snapshot use the calendar window. The console shows the two side by side, because a release manager asking "why does it say exhausted" needs to see which window is binding.

Options weighed

• ChosenPublish both; rolling drives the verdict: Two projections to keep fresh and a reader who must know which is which; no reset to game and a reportable month alongside
• RejectedRolling only: Cleanest incentive; leaves the business with no monthly figure and makes compliance reporting a derived estimate
• RejectedCalendar only: Simplest to explain and to seal; creates the end-of-month cliff and the first-of-month release rush
• RejectedPublish both and let each service choose which binds: Maximum local autonomy; makes cross-service comparison meaningless and gives every team an argument to have during an incident

Consequences

What it buys

• No reset date to hold changes against, so the freeze cannot be waited out
• The business keeps a monthly attainment figure that lines up with every other monthly report
• An incident's cost visibly decays, which makes the freeze feel like a constraint rather than a punishment

What it costs

• Two figures for the same SLO, and a reader who confuses them will reach the wrong conclusion
• A single bad interval can hold a team for four weeks, which is a real organisational cost and the main source of pressure for an override (ADR-11)
• Twice the materialisation work and twice the recompute on a definition change

Choose differently when

If the organisation's reliability commitments were contractual and monthly — credits owed on a calendar month — the calendar window would have to be canonical, because the figure that carries money cannot be the secondary one. In an internal engineering context the rolling window's incentive properties dominate.

Why it holds up over time

The arithmetic of both windows is permanent and the choice between them is organisational. What should survive is the structural answer: compute both from the same counters, declare one canonical, and name the window in every response. A platform that publishes one figure without saying which window it is will be misread forever.

Should the platform accept a definition it knows it cannot measure meaningfully?

Context

Three definitions arrive routinely and all three are worse than useless once published. An objective of 99.999% over a calendar month allows 26 seconds of failure, which minute-grain buckets cannot resolve — the resulting figure is noise presented as precision. A definition that does not say which status classes, synthetic traffic, health checks and client-cancelled requests are excluded has not actually defined anything, and the ambiguity will be resolved later by whoever is under pressure. A definition with unbounded label dimensions is a cardinality incident in the aggregate store, which is the dominant cost driver and the usual cause of a time-series platform becoming unaffordable. In all three cases the cheapest moment to object is before the definition exists.

Decision

Admission runs in the reconciler, before anything enters the registry, and refuses three classes of definition: an objective whose budget is finer than the measurement resolution can resolve, a definition that leaves its valid-event denominator implicit, and label dimensions exceeding a declared cardinality limit. A refusal fails the merge with an explanation; it is not a warning.

How it is realised on AWS

The reconciler computes the implied budget in events and minutes against the window length and bucket resolution, and refuses when the budget falls below the resolvable floor. The schema requires explicit inclusion or exclusion for each status class and traffic type, so an omission is a parse failure rather than a default. label_dims is a bounded list with a per-SLO cell-count cap that also bounds the outcome cube from ADR-03. All three checks run as part of the pull-request validation, so the feedback arrives in review rather than after merge.

Options weighed

• ChosenRefuse at authoring time, as a merge failure: Friction at the one moment it is cheap; teams will occasionally be blocked from expressing something they wanted
• RejectedAccept and warn: No friction; the warning is read once and the meaningless figure is published for a year
• RejectedAccept, and annotate the resulting figures as low-confidence: Honest in principle; nobody reads the annotation and the figure circulates without it
• RejectedRefuse only the cardinality case: Protects the cost model, which is the self-interested subset; leaves the two that make the figures wrong

Consequences

What it buys

• No SLO in the estate carries an objective the data cannot support
• The denominator — where an SLO is really defined and later gamed — is explicit for every SLO by construction
• Storage cost stays bounded per SLO and attributable to its owner

What it costs

• A team wanting a very high objective is blocked and may feel the platform is obstructing a legitimate goal
• The resolution check is arithmetic on assumptions; a change in bucket granularity changes which definitions are admissible
• Review-time refusal pushes the conversation into a pull request, where the reasoning is less visible than in a design discussion

Choose differently when

If bucket granularity moved from minutes to seconds, the resolvable floor would drop by nearly two orders of magnitude and the first refusal class would mostly disappear — at sixty times the bucket volume. The denominator and cardinality refusals are independent of resolution and would stand regardless.

Why it holds up over time

Refusing to represent what you cannot measure is a permanent property of honest measurement systems. The specific thresholds follow the storage design; the principle that admission is a gate rather than a warning is what stops the registry filling with definitions nobody can defend.

Can the read path afford the arithmetic the verdict needs?

Context

Computing the budget on every query is the correct-by-construction option: it is always consistent with the current definition, a definition change takes effect instantly, and there is no invalidation logic to get wrong. It is also a scan over up to 40,000 minute buckets per query, and the verdict API is specified at a p99 of 150 ms while absorbing 2,000 requests a second for two minutes during a coordinated release window. Those two facts do not reconcile. Incremental materialisation gives constant-time reads and introduces a staleness window plus invalidation logic that has to stay correct across late data, approved exclusions and retroactive definition changes — three things that all move history.

Decision

Budget state is a materialised projection advanced by a watermark over closed minute buckets. Reads are constant time. The projection carries its watermark, every response carries the resulting computation timestamp, and the verdict API returns insufficient-data once staleness exceeds a declared ceiling rather than serving a figure it cannot date. Invalidation is explicit: late data within the horizon, an approved exclusion and a new definition version each enqueue a scoped recompute (ADR-15, view 15).

How it is realised on AWS

The budget calculator consumes closed buckets from Data Explorer and writes summed counters plus derived percentages into Azure Cache for Redis, keyed by SLO and window kind, with the watermark minute alongside. The verdict API reads Redis only — it never touches the aggregate store on the request path, which is what makes the read tier independently available (ADR-16). Recompute runs in a separate deployment on preemptible capacity so a full-estate backfill cannot consume the capacity serving verdicts.

Options weighed

• ChosenIncremental materialisation with a watermark and a staleness ceiling: Constant-time reads and an honest staleness figure; invalidation logic that must survive late data, exclusions and version changes
• RejectedRecompute on read: Always consistent and trivially correct; cannot meet 150 ms p99 at 2,000 rps over a 40,000-bucket scan
• RejectedRecompute on read with a short-lived response cache: Looks like a compromise; produces the same staleness problem with none of the watermark's visibility into how stale
• RejectedMaterialise only the rolling window, recompute calendar on read: Halves the projection work; makes the monthly report path a tail-latency risk during exactly the reporting peak

Consequences

What it buys

• The read path is cheap, flat and independent of history length, so the gate's latency budget is unaffected by retention
• Staleness is a published number rather than an unknown, and the ceiling turns it into a typed refusal
• Recompute is a separate concern on separate capacity, which is what makes backfill affordable

What it costs

• Invalidation is the most intricate logic in the platform, and a missed invalidation serves a stale figure that looks authoritative
• A definition change is no longer instantaneous: it is a recompute with a visible duration
• The projection is one more thing that can be wrong in a way the aggregates are not, which is why shadow verification exists (view 15)

Choose differently when

If the verdict read rate were low — a handful of queries a day from a human rather than one per deployment from a machine — recompute-on-read would dominate on correctness and simplicity, and the whole projection could be deleted. The decision is driven entirely by the gate being a machine on the request path.

Why it holds up over time

The specific store will change. What survives is the pairing: if you cache a derived figure, publish its watermark and refuse to serve past a declared staleness ceiling. A cache without a visible age is the mechanism by which correct systems start giving confidently wrong answers.

What alerting signal has a defensible relationship to the objective, and how many rules can 1,200 SLOs carry?

Context

Threshold alerting on error percentage is what most organisations have, and it bears no relationship to any commitment: it fires on a bad minute regardless of whether the budget is threatened, and misses a slow leak that quietly exhausts a month. Burn rate fixes the relationship — consumption per unit time, normalised so 1× exhausts the budget exactly at window close — but a single burn-rate rule has to choose between catching a fast outage and catching a slow leak, and cannot do both. The canonical answer is several window and multiplier pairs evaluated simultaneously, which multiplies the rule count across 1,200 SLOs and multiplies the false-positive surface with it. Precision, not recall, is what determines whether pages are still answered in six months.

Decision

Each SLO carries exactly two rules, generated from a template rather than authored per service: a fast pair intended to page, and a slow pair intended to raise a ticket. Both require a short confirmation window to agree with the long window before firing. Page-class and ticket-class alerts route to different destinations, and a slow burn never pages. Every evaluation is recorded with its inputs.

How it is realised on AWS

Rules are compiled from the registry into an alert_rule row per SLO per class — fast at 14.4× over a one-hour long window with a five-minute confirmation window, slow at 3× over six hours — and evaluated by the burn-rate evaluator on separate ticks. Firing decisions go to Azure Monitor action groups, grouped by common dependency with a declared cap on simultaneous pages. alert_decision rows retain the evaluated inputs for 13 months, which is what makes page precision computable against incident records rather than estimated.

Options weighed

• ChosenTwo generated pairs per SLO, page and ticket, with confirmation: Catches both failure shapes with a bounded rule count; less tuning freedom than per-service authoring
• RejectedA single fast burn-rate rule: Half the rules and half the noise; misses the slow leak that exhausts a budget over a week, which is the failure nobody notices
• RejectedThe canonical four-window pairing: Best recall; 4,800 rules across the estate and a false-positive rate that degrades the precision the pages depend on
• RejectedPer-service authored rules: Maximum local fit; 1,200 hand-written configurations drift immediately and cannot be changed estate-wide

Consequences

What it buys

• A page means the budget is genuinely going, not that a minute looked bad
• Alerting behaviour across 400 services can be changed in one merge, which is the only way the configuration stays coherent
• Page precision is a measured property of the platform rather than an opinion held by the on-call rotation

What it costs

• Two rules miss failure shapes a four-window pairing would catch, and that gap is accepted deliberately
• Generated rules fit no service perfectly, so there will be SLOs whose thresholds are visibly wrong for their traffic shape
• Grouping by common dependency can swallow a genuinely separate incident that happens to share one

Choose differently when

If the on-call rotation were large enough that page volume were not the binding constraint, the four-window pairing's recall would be worth its false positives. At twelve people carrying 1,200 SLOs it is not — and the moment precision drops far enough that pages are routinely ignored, the platform has failed regardless of how many real failures it detected.

Why it holds up over time

Burn rate as the signal is arithmetic and permanent. The rule count is an organisational constraint that will move with the size of the rotation, and the template is the mechanism that lets it move without 1,200 edits. Recording the inputs of every decision is what will still be useful in ten years, when nobody remembers why a threshold was chosen.

Should the reliability authority be able to stop a deployment?

Context

If the platform holds the budget and the policy, the shortest path to a real freeze is for the platform to enforce it: call the deployment API, revoke the pipeline's credential, fail the check itself. It removes an integration and makes the policy unambiguous. It also puts this platform on the critical path of every deployment in the organisation, including the rollback that would end an outage, and it makes a false positive in SLI data an estate-wide inability to ship. The alternative is to publish a claim and let the deployment system act on it, which keeps the authority off the critical path but means the policy is only as real as the gate's willingness to honour it.

Decision

The platform's output is a signed, typed, time-limited verdict document — healthy, warning, exhausted or insufficient-data — carrying the figure, the definition version, the computation timestamp and the data coverage. The platform blocks nothing, revokes nothing and calls no deployment API. Enforcement is the deployment system's act, performed by reading the verdict and applying its own declared policy.

How it is realised on AWS

The verdict API returns a document signed with a non-exportable key in Azure Key Vault Managed HSM, with valid_until set five minutes ahead. The gate authenticates with workload identity federation, verifies the signature itself, and applies the policy recorded against the SLO. Signature validity is short precisely because a signed document is otherwise replayable forever, and the gate's own verification is what makes a cached verdict safe to use during an outage of this platform (ADR-13).

Options weighed

• ChosenPublish a signed verdict; the gate enforces: One integration and a policy that depends on the gate honouring it; the authority stays off the critical path of the release path
• RejectedThe platform calls the deployment API to block: Unambiguous enforcement; makes a false positive here an estate-wide inability to ship, including the rollback
• RejectedThe platform acts as the CI check itself: Neat integration story; the platform's availability becomes the pipeline's availability for every repository
• RejectedUnsigned verdict over mutual TLS: Simpler and adequate in transit; a cached verdict is then unverifiable, which removes the fail-static option entirely

Consequences

What it buys

• An outage of the reliability authority is not simultaneously an observability outage and a deployment outage
• The fail-open or fail-closed decision sits with the system that knows what it is deploying
• A verdict can be cached, archived, attached to a release record and verified later by anyone

What it costs

• The policy is only as strong as the gate's implementation, and a bypassable gate makes the arithmetic advisory in fact
• Signing is on the synchronous read path, so HSM latency and availability are inside the verdict API's own budget
• Two systems must agree on the verdict schema, and a schema change is a coordinated release

Choose differently when

If deployments were centralised in one pipeline owned by the same team as this platform, direct enforcement would be a smaller risk and would remove a real integration cost. Across 400 services and multiple pipelines it is not, and the rollback argument holds regardless of ownership.

Why it holds up over time

"Publish, do not enforce" is the critical design decision restated at the API boundary, and it will be challenged every few years on the grounds that the integration is tedious and the policy feels weak. The counter-argument does not change with scale: an authority on the critical path of the thing it governs cannot fail independently of it.

What does the release gate do when the platform cannot be reached?

Context

Having decided that the gate enforces (ADR-12), the unreachable case needs an answer, and all three available answers are bad. Fail open evaporates the policy during exactly the incident most likely to take both systems down — a regional event, a shared dependency, the 90-minute recompute window after a failover. Fail closed makes this platform a single point of failure for every deployment in the organisation, including the rollback that would end the outage, which inverts the platform's purpose precisely when it matters. Caching the last verdict and continuing to enforce it turns the problem into a staleness ceiling, which is a judgement with no obviously correct value but at least degrades along a measurable axis.

Decision

The gate caches the last signed verdict it received and, when the platform is unreachable, continues to enforce that verdict up to a declared staleness ceiling, after which it fails open and records that it did so. Separately and unconditionally, changes classed as reliability fixes or rollbacks are exempt from a freeze by default — a freeze that blocks the fix for the outage that caused it is a defect, not a policy.

How it is realised on AWS

The policy object carries gate_mode and an unreachable behaviour, and the verdict's signature plus valid_until let the gate verify a cached document it has held for hours. Rollback and hotfix classes are declared in the policy's exempt_classes and evaluated by the gate from the change's own metadata, not from anything this platform knows. Bypasses and ceiling expiries are reported as first-class figures next to the override register, so a rising bypass rate reads as a policy failure rather than a compliance one.

Options weighed

• ChosenFail static on a verified cached verdict, with a ceiling and fixes exempt: Degrades along a measurable axis; the ceiling is a judgement nobody can derive, and the policy weakens with age
• RejectedFail open immediately: Never blocks a fix; removes the policy during the correlated outage that is the likeliest reason the platform is unreachable
• RejectedFail closed: Strongest policy; makes the reliability platform able to stop the rollback that would end an incident
• RejectedFail closed with a documented manual override: Looks balanced; puts a manual step into the incident path at the moment humans are least able to execute one

Consequences

What it buys

• An outage of this platform never prevents a rollback or a reliability fix
• The policy survives a short outage intact rather than evaporating on the first timeout
• Degradation is observable: bypasses and ceiling expiries are counted and reviewed

What it costs

• A gate enforcing an eight-hour-old verdict is enforcing history, and no ceiling value is obviously correct
• The gate must implement signature verification and a cache, which is real work in every pipeline that integrates
• Exempting fixes depends on change classification the gate performs, which can be claimed falsely

Choose differently when

If this platform's measured availability were materially better than the deployment system's own, fail closed would stop being a meaningful risk and would give a stronger policy for free. It would still need the rollback exemption, which is the part that is not about availability at all.

Why it holds up over time

Fail-static is the standard answer for a control plane whose consumers must keep working without it, and it will outlive the specific ceiling. The rollback exemption is the part most often forgotten and most expensive to learn during an incident; it should be treated as a permanent invariant rather than a configuration choice.

Who is allowed to decide that a bad forty minutes did not count?

Context

Some intervals genuinely should not count: a dependency's declared maintenance, a load test run against production, a measurement artefact already diagnosed. Refusing all exclusions sounds principled and is not — the pressure does not disappear, it relocates, and the next move is to widen the valid-event denominator instead, which is the same adjustment made invisibly. So exclusions have to exist, which creates a privilege whose holder can make any breach appear met. Three mechanisms are available: delete or ignore the interval, annotate it with the original retained, or leave the breach intact and grant a compensating credit. The second preserves the measurement; the third preserves it too but complicates every figure with a second adjustment nobody reads.

Decision

An exclusion is an overlay, never an edit: the interval is annotated with a reason code, a named approver, an expiry and a permanent audit record, and the buckets are untouched so the unexcluded figure stays retrievable forever. The author of an SLO cannot approve an exclusion against it, an exclusion beyond 60 minutes requires a second approver, and exclusions have no permanent form — they expire by default at 30 days.

How it is realised on AWS

exclusion_window rows in Azure SQL carry the interval, reason_code, two approver fields and expires_at; sli_bucket is never modified. The derivation applies the overlay at window reduction, so both figures are a query away. The privileged API refuses an approval whose actor matches the definition's author, writes the audit ledger entry before the registry write, and counts every exclusion into a register reported weekly alongside attainment. Audit rows are Azure SQL ledger tables, so an after-the-fact edit is detectable rather than merely prohibited.

Options weighed

• ChosenAnnotate, with separation of duties and an expiry: Both figures kept and the adjustment visible; a privilege that still exists and can still be abused by two colluding people
• RejectedNever exclude anything: Maximally honest in appearance; relocates the pressure into denominator definitions, where the same adjustment is invisible
• RejectedDelete or ignore the excluded buckets: Simplest to implement and to read; destroys the ability to answer what the month looked like before anyone intervened
• RejectedBudget credits instead of exclusions: Preserves the measurement perfectly; every figure then carries a second adjustment, and the compound number is one nobody trusts

Consequences

What it buys

• The question "what did this month look like before the adjustments" always has an answer
• The privilege to improve a figure never sits with the party being measured
• A routine exception becomes visible as a rising count rather than as an unremarkable habit

What it costs

• Two colluding approvers defeat the control entirely, and nothing in the architecture detects collusion
• Every published figure now has an excluded and an unexcluded form, and a reader who quotes the wrong one is misleading without intending to
• Exclusion approval is operational work for the reliability lead at exactly the moments they are busiest

Choose differently when

If the published figure ever carried contractual weight — customer credits, a regulatory filing — internal separation of duties would stop being sufficient, because the measured party and the approver are both inside the same organisation. At that point the figure needs an external witness: a notarised snapshot or third-party attestation.

Why it holds up over time

Separation of duties on an adjustment privilege is structural and survives reorganisations in a way that norms about who may adjust a number do not. The specific thresholds will move; the rule that an author cannot approve their own exclusion is the part worth defending.

If last quarter's attainment is quoted in a board pack, can it change afterwards?

Context

Everything else in this platform is deliberately recomputable: a corrected definition yields a corrected history (ADR-05), an approved exclusion changes the current figure (ADR-11), late data moves a bucket within the horizon (ADR-08). That is correct for an operational figure and wrong for a compliance record. A monthly report regenerated from current state will quietly produce a different answer every time it is run, which makes it useless as evidence — and the circumstances under which it changes are exactly the circumstances under which someone would want it to.

Decision

At the close of each calendar window, a snapshot of the figure — total, consumed, remaining, coverage and the definition version in force — is sealed in write-once storage with no update or delete path. Subsequent definition changes, exclusions and recomputes produce a new current figure and cannot alter a sealed snapshot. The monthly report is rendered from snapshots, never from current state.

How it is realised on AWS

budget_snapshot rows are written once and are immutable in the schema; the rendered snapshot document is written to Azure Blob Storage under a time-based immutability policy with a five-year retention, alongside audit ledger entries in Azure SQL ledger tables. The window manager performs the seal; no other component has write access to that container. A sealed figure that is later found to be wrong is corrected by publishing a dated correction next to it rather than by editing it.

Options weighed

• ChosenSeal an immutable snapshot at close; correct by addition: A record that means something; a permanent discontinuity in the trend when a long-standing definition error is found
• RejectedRegenerate reports from current state: Always reflects the best current understanding; a figure that changes between readings cannot be evidence
• RejectedSnapshot, but allow a privileged correction: Handles the genuinely-wrong case; the correction privilege is exactly the one an interested party would want
• RejectedSeal with a short grace period before immutability: Pragmatic compromise; the grace period becomes the window in which inconvenient months are revised

Consequences

What it buys

• A quoted historical figure is stable, dated and attributable to the definition in force at the time
• An auditor's question is answerable without trusting the platform's own access control
• The operational figure can stay freely recomputable, because it is no longer doing the compliance job

What it costs

• A definition error found after a close leaves a permanent discontinuity between the sealed figure and the corrected trend
• Write-once retention is irreversible by design: a retention policy set wrongly cannot be corrected, with a five-year blast radius
• Two figures exist for every past month — the sealed one and the current recomputation — and the difference needs explaining

Choose differently when

If the figure were purely internal and never quoted outside engineering, regenerating from current state would be simpler and the loss would be theoretical. The moment a number appears in a board pack or a customer conversation, it needs to be the same number next quarter.

Why it holds up over time

Append-only evidence with correction-by-addition is how accounting has worked for centuries and how every durable audit system works. It will outlive the storage product; the thing to preserve is that the component which seals is the only one with write access, and that nothing has an update path.

What is actually worth backing up, and can two regions compute the same budget?

Context

The platform holds four kinds of state with very different properties. The registry is irreplaceable and small. The minute buckets are irreplaceable, large and replayable from quarantine. The sealed snapshots and audit ledger are evidence with no update path. The budget projection, compiled rules and report extracts are none of these: they are a deterministic function of the first two, and a backup of them is a slower way to obtain a staler version of something that can be recomputed. Separately, there is a question about regions: a second write region would improve availability and would also have two independent computation planes deriving the same budget from partially overlapping, independently late bucket sets — producing two figures that no key reconciles, in a system whose entire purpose is to produce one.

Decision

Derived state is not backed up at all; its recovery objective is stated as recompute time — a trustworthy projection within 90 minutes of regional recovery. The registry and audit ledger are backed up with a one-minute RPO and geo-replicated; aggregates are replicated and replayable; snapshots are write-once with cross-region redundancy. There is exactly one write region. Failover is promote-then-recompute, and the honest intermediate state is insufficient-data (ADR-04).

How it is realised on AWS

Azure SQL is zone-redundant with point-in-time restore and an active geo-replica in the secondary; the Data Explorer cluster is followed in the secondary; snapshots and audit sit in read-access geo-redundant immutable storage. Redis holds the projection and is not backed up — the recompute runner rebuilds it from aggregates plus registry. The secondary region's compute is scaled to zero. A shadow rebuilder recomputes sampled SLOs continuously and alerts on divergence, because a recovery path exercised only during an incident is a claim rather than a capability.

Options weighed

• ChosenOne write region; rebuild derived state rather than restore it: One answer always, and a recovery story that improves as compute gets cheaper; 90 minutes of insufficient-data after a regional loss
• RejectedTwo active write regions: Best availability; two computation planes produce two irreconcilable budgets for the same SLO
• RejectedOne write region, with the projection backed up for fast restore: Shortens the gap; restores a figure that is stale by exactly the backup interval, which the verdict would have to disclose anyway
• RejectedActive-passive with synchronous projection replication: Near-zero gap; couples the read path's latency to cross-region replication for state that is disposable by construction

Consequences

What it buys

• There is never more than one budget figure for an SLO
• Backup scope is small, so restore drills are cheap and actually get run
• The recovery story improves automatically as compute gets faster, where a restore of growing data gets slower

What it costs

• A 90-minute window of insufficient-data after a regional loss, during which every gate falls back to its unreachable behaviour (ADR-13)
• The secondary's capacity on promotion is a cold-start assumption until a game day proves it
• Recompute capacity has to be available during a regional incident, which is when it is most contended

Choose differently when

If the verdict API ever became a hard dependency of something that cannot tolerate 90 minutes of insufficient-data — an automated capacity controller, a customer-facing commitment — a second write region would become necessary, and the reconciliation problem would have to be solved rather than avoided, probably by partitioning SLO ownership by region.

Why it holds up over time

Classifying state by rebuildability rather than by importance is the decision that keeps paying off: it shrinks the backup surface, makes recovery testable, and improves with hardware. The one-write-region constraint is specific to needing a single answer and would only change if the arithmetic could be partitioned.

Who notices when the thing that notices everything stops working?

Context

The platform computes SLOs, so the obvious way to monitor it is to give it SLOs of its own and compute them with the same engine. That is useful and insufficient: the measurement is circular, and the failures that matter most are exactly the ones that take the engine down with the signal. A platform that is up, fast and serving stale or wrong figures passes every internal check, and a platform that is entirely down produces no alert at all because the thing that would produce it is the thing that is down. Nothing inside the system can close that gap, and a self-check deployed inside the same region and the same subscription shares enough failure modes to be nearly as blind.

Decision

The platform holds SLOs on itself — budget freshness, verdict availability, alert detection latency — computed by the same engine for day-to-day operations. Independently, a deliberately minimal watchdog deployed in a third region probes the verdict API from outside and alerts on absence and on staleness, with no dependency on any component it is watching. The package documents that the platform is not the sole authority on its own availability.

How it is realised on AWS

An Azure Functions app in a third region, on its own subscription and its own alerting path, calls the verdict API on a fixed interval with a synthetic read-only SLO, verifies the signature, and checks the computation timestamp against its own clock. It shares no store, no identity and no deployment pipeline with the platform. It raises to the platform rotation through an action group that does not traverse the primary region. Its own availability is explicitly not measured by this platform, which bounds the recursion at one level.

Options weighed

• ChosenSelf-measured SLOs plus a minimal independent watchdog elsewhere: Bounds the circularity for the cost of a second deployment; the recursion is stopped by fiat rather than solved
• RejectedSelf-measured SLOs only: No extra component; produces no signal in the one failure mode where a signal is essential
• RejectedA full second instance of the platform watching the first: Rich signal and genuine independence; doubles the operational surface and raises the same question one level up
• RejectedRely on the consumers to notice: Zero cost; the gate failing static is silent by design, so nobody notices until a release is questioned

Consequences

What it buys

• Total unavailability of the platform produces an alert from outside it
• Staleness is checked against an independent clock, so a stuck computation plane serving old figures is caught
• The watchdog is small enough to be obviously correct, which is the only useful property for a component of this kind

What it costs

• A second deployment, subscription and alerting path to maintain, which will be neglected because nothing ever happens to it
• The watchdog's own availability is unmeasured, so the recursion is stopped by assertion
• A correlated failure of platform and watchdog is undetected until a human notices

Choose differently when

If an external synthetic monitoring service were already trusted for estate-wide availability checks, it would be the better watchdog — genuinely outside the organisation's cloud estate, and already staffed. Building one is only justified because the probe needs to verify a signature and inspect a computation timestamp rather than just receive a 200.

Why it holds up over time

The need for an external observer of an observer is permanent and the answer is always the same shape: something small, something elsewhere, something with no shared dependencies, and an explicit decision about where the recursion stops. The implementation will be replaced; the reasoning will not.