Architecture Views

When a caller needs an endpoint and the discovery system is unreachable, what is the correct behaviour?

Context

The obvious design makes the registry the single truth and has callers consult it, or hold a short lease from it, per resolution. It gives one answer that is always current, and it means every one of twelve million internal requests a second depends on one distributed system being available. That system's availability then becomes the ceiling on every product's availability, and its worst day is everybody's worst day. The alternative is to accept that routing runs on a view that may be stale, and to make staleness the normal operating condition rather than an error — which moves the hard problems into the client and makes propagation delay the number the design lives or dies by.

Decision

The view a client already holds is authoritative for routing. The control plane computes and publishes improvements to that view and is never consulted synchronously on a request. Every client persists its last-known-good view locally and continues routing from it, unchanged, through a total control-plane outage for at least 60 minutes, degrading only by staleness. A resolution that cannot be served from cache or stream fails to a documented default; it never blocks.

How it is realised on AWS

Envoy sidecars hold incremental xDS subscriptions against the regional control plane on EKS and write each applied view to a local volume. On stream loss the sidecar keeps serving the persisted view and re-subscribes with jittered backoff. The thin resolver library used by sidecar-less callers implements the same cache-first contract. NLB and ALB target groups are synchronised from the same views, so edge-routed traffic inherits the behaviour without taking a dependency of its own.

Options weighed

• ChosenClient-held last-known-good view is authoritative; control plane advisory: Decouples caller availability from control-plane availability entirely; pays in staleness and in guards that must exist twice.
• RejectedRegistry consulted or leased per resolution: Exactly consistent and makes the discovery plane a hard dependency of every request in the company.
• RejectedShort-TTL DNS only, no persistence: Looks like caching but is not: a resolver that honours a 5 s TTL has no view at all 5 s into an outage.
• Right elsewhereCentral routing proxies holding the view on the caller's behalf: Right where client libraries cannot be changed; adds a network hop and a new tier that must itself be more available than the control plane.

Consequences

What it buys

• The control plane can be deployed, patched, scaled and drilled like an ordinary service, because its unavailability is a staleness event rather than an outage.
• A registry failure, a propagation-tier deploy and a cross-region partition all collapse into the same bounded failure mode.
• Resolution latency is an in-process lookup at p99 ≤ 1 ms, which removes discovery from every latency budget in the estate.

What it costs

• Every safety mechanism that bounds a wrong decision must exist in the client as well as the control plane — the shrink cap, the ejection cap, the refusal of an empty set.
• Correctness now depends on the least-updated client library in the estate, which is a fleet-management problem rather than an architecture one.
• Withdrawal propagation delay becomes the single most important number in the design, and the only honest measure of how wrong the data plane is permitted to be.

Choose differently when

If the estate were small enough that one control plane could credibly be more available than every service calling it, or if routing to a withdrawn endpoint were unrecoverable rather than retryable — a payment captured twice, an irreversible write — then exact consistency is worth the dependency and the registry should be the authority. The decision should also be revisited if client library heterogeneity becomes unmanageable, because this design silently delegates safety to those libraries.

Why it holds up over time

This is a statement about where authority lives, not about a protocol. It survives replacing xDS, Envoy, the orchestrator and the registry store, because none of those changes which copy of the truth is allowed to route.

Which of the platform's own data has to survive a total loss of the control plane, and which should be thrown away rather than restored?

Context

A discovery plane holds three quite different kinds of data and the temptation is to treat them alike. Service definitions, instance rows, health contracts and policy are small, exact and irreplaceable: nothing can recompute who owns a service or what its readiness contract says. Probe results and caller outcomes are enormous, arrive at over a million a second, and matter only in aggregate over the last few minutes. Eligibility decisions and published views are derived from the other two by a pure function. Storing the third class durably invites repairing it, and a platform that repairs its routing state in place has no recovery story at all.

Decision

Desired state is the system of record with RPO 0, held in a replicated store that is strongly consistent within a region. Observed state is recomputable with RPO 60 s, held in a store sized for write volume rather than durability, where losing a window of samples reduces confidence without corrupting a verdict. Computed state — eligibility, weights and versioned views — is held in memory only, has no RPO, and is rebuilt rather than restored. Every eligibility transition and policy change is additionally emitted to a durable append-only log for audit and incident reconstruction.

How it is realised on AWS

Registry and policy on Aurora PostgreSQL across three AZs. Observed state in DynamoDB with a 7-day TTL and on-demand capacity sized for 1.2 M writes a second. Eligibility and assembled views live in the control-plane pods' memory and are reconstructed on start from Aurora plus the last few minutes of DynamoDB samples. Transitions stream to Kinesis and land in S3 with Object Lock for the 5-year audit retention.

Options weighed

• ChosenThree state classes with three durability contracts: Makes RTO a rebuild throughput question and removes any temptation to repair routing state.
• RejectedOne durable store for everything: Simplest to operate and makes the sample write rate the sizing constraint for the store holding the service catalogue.
• RejectedPersist computed eligibility for faster restart: Buys seconds of startup time and reintroduces repairable routing state, which is the failure mode being designed out.
• DeferredEvent-sourced control plane with a replayable decision log: Attractive for reconstructing exactly what the platform believed at a past instant; the transition log gives most of it without the operational weight.

Consequences

What it buys

• RTO 5 minutes is a rebuild measurement rather than a restore hope, and the rebuild is exercised on a schedule instead of during an incident.
• A scoring or fusion bug is repaired by recomputing from retained samples, with no state surgery.
• The sample store can be cheap and lossy, which is what makes a million writes a second affordable.

What it costs

• The rebuild path must be fast enough to meet the RTO at full scale, which is a continuing throughput commitment and not a one-off test.
• Losing up to 60 s of samples briefly reduces confidence in eligibility, which interacts with the unknown state and has to be reasoned about together.
• Two stores with different consistency models mean two operational runbooks.

Choose differently when

At a tenth of this scale the sample volume no longer forces a separate store, and one strongly consistent store for everything is simpler and defensible. Persisting computed state becomes right if control-plane restarts ever need to be measured in seconds rather than minutes — for example if the platform were ever placed on the request path, which ADR-01 forbids.

Why it holds up over time

The classification by what-happens-if-lost outlives any particular store. Swapping Aurora, DynamoDB or Kinesis changes nothing about which data may be thrown away.

How does the platform learn that an instance exists?

Context

The classical service-discovery pattern has each process register itself on start and deregister on stop. It is simple, it works in a demo, and it has two failure modes that look nothing alike. A process that crashes before deregistering leaves a zombie endpoint that is only removed when its lease expires. A process that starts but fails to register is worse: the fleet looks healthy and is short of members, and nothing in the system is in a position to notice, because the only thing that knew the instance should exist was the instance. Meanwhile the orchestrator already holds exactly this information, as reconciled desired state, for the majority of the estate.

Decision

For orchestrated workloads, instance existence is derived by continuous reconciliation against the orchestrator's own state; the workload does not register itself. An authenticated registration API exists only for runtimes nothing watches. Every instance receives a registry-unique identity that is stable for its lifetime and never reused, so a restarted process is a new instance rather than a continuation. Ungraceful termination is caught by lease expiry within 30 s; graceful termination deregisters before the process stops accepting connections.

How it is realised on AWS

A reconciler watches EndpointSlices across 12 EKS clusters and ECS task state change events, and consumes EC2 Auto Scaling lifecycle hooks for VM fleets. Divergence between orchestrator state and registry state is reported as a reconciliation diff, which is also how registration leaks are found. The registration API is reserved for declared external endpoints and for fleets outside Auto Scaling, authenticated by instance profile or IRSA.

Options weighed

• ChosenReconcile from orchestrator state; explicit registration for the remainder: Makes a missing instance detectable as a diff rather than invisible, and removes a whole class of startup failure.
• RejectedSelf-registration by every workload: One mechanism for the whole estate, at the cost of a failure mode where the platform cannot know what it has not been told.
• RejectedSidecar registers on the application's behalf: Moves the problem into the sidecar without removing it; the sidecar can also fail to register.
• Right elsewhereOrchestrator-native discovery only, no separate registry: Right for a single-cluster estate; it cannot represent a VM fleet, a managed function or a third-party endpoint, which is most of the hard part here.

Consequences

What it buys

• A missing instance appears as a reconciliation diff instead of as a quietly smaller fleet.
• Non-reusable identity makes a stale cached endpoint detectably stale rather than ambiguously reassigned.
• The registration API's attack surface shrinks to the small non-orchestrated tail.

What it costs

• One watch integration per runtime type, each with its own failure modes and its own back-pressure behaviour.
• Reconciliation lag becomes a signal that has to be watched, because it is now the path by which existence is learned.
• A heterogeneous estate ends up with two registration mechanisms, which has to be documented so callers do not care.

Choose differently when

If the entire estate ran on one orchestrator with one API, reconciliation and native discovery converge and a separate registry earns less. Self-registration becomes the better choice if workloads routinely run somewhere nothing watches — a developer laptop, a customer-managed environment — where there is no authority to reconcile against.

Why it holds up over time

The principle is to prefer the authority that already knows over the party that would have to be asked. That holds as orchestrators change.

Who should be believed about whether an instance can serve: the instance, a prober, or the callers?

Context

Each source is wrong in a different and characteristic way. A self-report knows the most about internal state and a broken process can report healthy with complete confidence, while a compromised one can simply lie. An external prober is independent, costs N×M traffic at sixty thousand instances, and measures a path no real caller uses — a replica can be perfectly healthy for the prober and broken for everyone else. The outcomes real callers observed are the only evidence that measures what actually matters, cost nothing because the traffic was happening anyway, and go silent on a low-traffic service and cannot distinguish a bad callee from a bad caller. Grey failure — brownout, exhausted pool, long GC pause — is assumed here to be the most common real failure, and it is precisely the class a probe cannot see.

Decision

The platform collects three separately addressable evidence classes — active probe, self-report, and passive caller-observed outcomes — and fuses them into one eligibility decision. The absence of one class is never read as a negative result from another. Where a service's traffic volume makes passive signal statistically sufficient, passive outcomes are the primary evidence and probing drops to a confirmation cadence; below that threshold probing remains primary and the service is reported as having single-class evidence.

How it is realised on AWS

Probers run as a per-AZ DaemonSet with bounded target assignment, so no instance is probed by every prober. Envoy sidecars report per-endpoint outcome counters to a Kinesis-backed ingest at 1.2 M results a second combined. Self-reports ride the Instance API under the workload's own identity. Fusion weights are configured per service, with the traffic threshold for passive-primary declared in the health contract.

Options weighed

• ChosenAll three classes fused, passive-primary above a traffic threshold: The only arrangement that sees grey failure, survives a prober-network partition, and still covers low-traffic services.
• RejectedActive probing only: Universal and uniform, and blind to exactly the failure mode that is most common and most user-visible.
• RejectedSelf-report only: Cheapest and most informed, and unfalsifiable — a confidently wrong process is indistinguishable from a healthy one.
• RejectedPassive observation only: Free and accurate where traffic exists; a quiet service becomes permanently unknown, which is not an answer.
• Right elsewhereCaller-side health checking with no central evaluation: Right for a small estate with homogeneous clients; at 450 services it means 450 differing opinions and no shared record.

Consequences

What it buys

• Grey failure is detectable, which is the difference between this design and one that only finds dead processes.
• A prober-network partition degrades confidence instead of draining a healthy fleet, because a second class still speaks.
• Probe traffic stays under 0.5% of internal bytes, because the expensive class is used where the free one cannot reach.

What it costs

• Three ingest paths, three failure modes, and a fusion function whose weights are another thing to get wrong.
• Per-service weighting is a configuration surface that most teams will leave at its default, so the default has to be right.
• Passive signal requires every client library to report outcomes, which re-couples correctness to client fleet management.

Choose differently when

If every service in the estate carried enough traffic for passive signal to be statistically sufficient, active probing becomes a confirmation mechanism that could arguably be dropped. If client libraries cannot be trusted to report outcomes, probe-primary with self-report corroboration is the honest fallback — and grey failure then has to be caught by the callers' own error budgets instead.

Why it holds up over time

The classification of evidence by how it can be wrong is independent of any mechanism. Whatever replaces probes and sidecars, the three ways of being wrong remain three.

What should the platform conclude when an instance stops producing any health signal at all?

Context

A two-valued health model has to map silence onto one of its two values, and both choices are wrong in an important case. Mapping silence to unhealthy means that the first time the observation path breaks — a prober network partition, an ingest outage, a bad agent rollout — the platform concludes that the entire fleet is dead and drains a service that is working perfectly. This is not a theoretical failure: the observation path is a separate system from the thing observed, and it has its own availability. Mapping silence to healthy is the opposite error and keeps dead instances in rotation indefinitely. Neither is a statement about the instance; both are statements about the observer.

Decision

Eligibility has four values: eligible, degraded with a weight, ineligible, and unknown. An instance whose signals have gone silent because the observation path failed is classified unknown and is not removed from rotation on that basis. Moving an unknown instance to ineligible requires corroboration from a second, independent evidence class — most often the caller-observed outcomes, which do not depend on the prober network. Lease expiry remains an independent path to removal for an instance the orchestrator also no longer reports.

How it is realised on AWS

The evaluator distinguishes 'no sample within the contract interval' from 'sample reporting failure', and tracks per-class silence separately. Prober reachability is itself a monitored signal, so a prober-side outage is attributable rather than being inferred from a thousand simultaneously silent instances. The unknown count per service is exported and alertable.

Options weighed

• ChosenFour-state eligibility with unknown as a distinct value: Separates a statement about the instance from a statement about the observer, which a boolean cannot do.
• RejectedSilence treated as unhealthy: Conventional, simple, and converts any observation-path failure into a self-inflicted outage of every service at once.
• RejectedSilence treated as healthy: Safe against observer failure and keeps dead endpoints in rotation until something else notices.
• DeferredSilence resolved by a quorum of probers: Reduces the chance of correlated observer failure without eliminating it; the second evidence class achieves more for less machinery.

Consequences

What it buys

• An ingest or prober outage degrades confidence instead of draining the estate, which is the single largest self-inflicted risk this design removes.
• The unknown count is a direct measure of observation coverage, which turns a blind spot into a reported number.
• Operators get a state that matches what they actually know, which makes an override decision better informed.

What it costs

• Unknown instances keep receiving traffic, so a genuinely dead instance may stay in rotation longer than a two-state model would allow.
• Four states multiply the paths through the evaluator and through every client that interprets a view.
• The design depends on a second evidence class existing — which is exactly what a low-traffic service lacks.

Choose differently when

If the observation path could be made more available than the workloads it observes — probing from within the same pod, for example, with no network between them — silence becomes genuinely informative about the instance and the unknown state earns less. For a service where serving a dead endpoint is more costly than serving none at all, mapping silence to unhealthy is defensible and should be a per-service choice.

Why it holds up over time

The distinction between 'I know it is broken' and 'I cannot see it' is permanent. Any monitoring or discovery system that collapses the two will eventually drain a healthy fleet.

If a service's database is degraded, should the service report itself unready?

Context

Reporting unready is the honest answer and it is what most readiness probes end up doing, because wiring a dependency check into a readiness endpoint is a two-line change that feels responsible. It is correct when the dependency is per-instance: that replica cannot work, others can, route around it. It is catastrophic when the dependency is shared, which it usually is. Every replica checks the same database, every replica fails the same check, every replica withdraws within the same few seconds, and a service that could still serve the eighty per cent of its requests that never touch that database serves nothing at all. The cascade then propagates outward, because the callers of that service now have no healthy endpoints and may withdraw in turn.

Decision

Dependency health is expressible at three authority levels, declared per service in the health contract: informational, weight reduction, or unreadiness. The default is weight reduction. Unreadiness from a dependency failure is permitted only where the dependency is declared non-shared across the service's replicas. The minimum healthy fraction remains the backstop regardless of the declared level, so even a misdeclared shared dependency cannot drain a service.

How it is realised on AWS

The health contract carries a dependency_authority field and a per-dependency shared flag. Self-reports name the dependency and the observed degradation; the evaluator applies the declared authority. A contract change that raises a shared dependency to unreadiness authority is rejected at the policy API rather than at runtime.

Options weighed

• ChosenWeight reduction by default, unreadiness only for declared non-shared dependencies: Keeps partial function available, and makes the dangerous configuration something a team has to assert rather than inherit.
• RejectedDependency failure always sets unreadiness: Honest per instance and a correlated-withdrawal generator per service; this is the single most common way a fleet drains itself.
• RejectedDependency health informational only: Safe against cascades and discards real information about which replicas are least able to serve.
• DeferredPer-endpoint rather than per-instance health: The right long-term answer — withdraw the routes that need the database, keep the rest — and it needs route-level health in the data plane, which is a larger change than this platform.

Consequences

What it buys

• A shared-dependency wobble degrades a service rather than removing it, which is almost always what the user would choose.
• The dangerous configuration is opt-in and policy-checked, so the default is the safe one.
• Partial function stays available, which matters because most requests to most services do not touch the degraded dependency.

What it costs

• Capacity stays in rotation that will fail the subset of requests that do touch the dependency, so callers see errors they could have avoided.
• Teams must classify their dependencies as shared or not, and will sometimes get it wrong.
• Per-instance granularity is a compromise: the honest unit is the route, and this design does not have it.

Choose differently when

Where a dependency is genuinely per-instance — a local cache, an attached volume, a sidecar — unreadiness is correct and should be declared. Where failing a request is unrecoverable for the user rather than retryable, withdrawing is better than serving, and the authority level should be raised deliberately with the fraction guard understood as the only remaining backstop.

Why it holds up over time

The distinction between shared and per-instance dependency failure is a property of topology, not of technology, and it will decide this question for as long as replicas share anything.

How many failures should it take to remove an instance, and how many successes to let it back?

Context

Every damping mechanism buys stability by paying detection latency, and the exchange rate is not obvious. Consecutive-failure thresholds are simple and multiply the detection budget by the threshold: a three-strike rule on a one-second probe makes a three-second detection a nine-second one. A decaying score responds to failure rate rather than to the last sample, which is closer to what matters and introduces tuning that nobody can reason about at three in the morning. Symmetric thresholds are the worst of the options: an instance that is marginal will oscillate, and an oscillating instance destabilises every client's view, which is worse than either verdict being applied consistently.

Decision

Eligibility is computed from a decaying score per instance per evidence class, with asymmetric thresholds: re-entry is strictly slower than removal. A hard failure — connection refused, process gone, lease expired — bypasses the decay entirely and removes immediately, because damping exists to absorb ambiguity and a closed port is not ambiguous. Damping is bounded: it may add no more than 5 s to the p99 detection budget for a genuine failure, and that bound is the constraint the score's parameters are fitted to.

How it is realised on AWS

The evaluator holds an exponentially weighted failure rate per instance per class, with configurable half-life, and a separate hard-failure channel that short-circuits to ineligible. Thresholds live in the health contract with platform-enforced floors. The 5 s damping budget is measured as the difference between first-failed-sample and published-withdrawal for the hard and soft paths respectively, and reported per service.

Options weighed

• ChosenDecaying score, asymmetric thresholds, hard-failure fast path: Responds to rate rather than to the last sample, and does not make an obviously dead instance wait for statistics.
• RejectedConsecutive-failure thresholds only: Trivial to explain and multiplies the detection budget by the threshold, which breaks the 10 s p99 commitment.
• RejectedSymmetric hysteresis: Simpler to configure and guarantees oscillation on marginal instances, which is the failure this is meant to prevent.
• DeferredMachine-learned anomaly scoring over the sample stream: Might catch failure patterns a rate misses; an unexplainable verdict in a routing decision is a worse trade than a missed subtlety.

Consequences

What it buys

• Marginal instances stop oscillating, so client views stay stable and the delta churn volume stays bounded.
• A dead process is removed in 10 s p99 regardless of the damping configuration, because it never enters the damped path.
• The damping cost is an explicit, measured 5 s rather than an emergent property of three thresholds multiplying.

What it costs

• The score's half-life is a parameter that few teams will understand, so the platform default carries most of the weight.
• A slowly-degrading instance crosses the threshold later than a consecutive-failure rule would, by design.
• Two paths through the evaluator — damped and fast — is two code paths that must agree about everything else.

Choose differently when

If the dominant failure mode in the estate turned out to be hard rather than grey, the decaying score earns little and consecutive-failure thresholds would be the simpler, more explicable choice. If detection latency ever becomes more valuable than stability for a specific service — a payment path, say — the thresholds should be tightened for that service and the resulting flapping accepted and quarantined.

Why it holds up over time

Asymmetry between removal and re-entry is a permanent property of any system where a wrong removal is cheaper to reverse than a wrong retention is to discover. The specific scoring function is replaceable.

What should happen when most of a service's replicas report unhealthy at the same moment?

Context

Correlated mass withdrawal is the characteristic failure of a competent health-checking system. It arrives from a bad deploy, a shared dependency outage, or — most often — a health-check configuration change that is wrong in a way nobody noticed, because health-check config is rarely treated as production code. The platform then does exactly what it was asked to do, removes almost every replica, and the service's availability goes to zero because its health checking worked. Honouring every withdrawal is the honest behaviour and it means a two-line YAML change can take down a service more reliably than deleting its deployment.

Decision

Each service carries a minimum healthy fraction per zone, defaulting to 50%. When the eligible fraction would fall below it, the platform disregards health and returns the full endpoint set, declares in the response that it has done so, and raises it as an alert. A removal that would take the last eligible instance of a service in a region raises an incident-grade signal rather than completing silently. The guard sits downstream of evaluation, so a bug in scoring or fusion cannot bypass it.

How it is realised on AWS

The guard is evaluated in the view assembler, per service per zone, after eligibility and before delta encoding. Views assembled under the guard carry a degraded-confidence marker that clients surface in their own telemetry, and the fraction-guard trip is one of the platform's four alarms.

Options weighed

• ChosenDisregard health below a per-service, per-zone fraction, and declare it: Converts a self-inflicted outage into a declared degradation, which is nearly always the better failure.
• RejectedHonour every withdrawal: Maximally honest and makes the health-checking system the most effective outage generator in the estate.
• RejectedA single global fraction for all services: One number to reason about, and wrong for both a three-replica service and a three-thousand-replica one.
• Right elsewhereRequire a human decision above a blast-radius threshold: Right where serving a known-bad instance is unsafe — a payment or data-mutating path — and wrong at three in the morning for a read path.
• DeferredGraduated response: reduce weights before disregarding health: A smoother curve and worth having; the cliff is easier to reason about during an incident, which is when it fires.

Consequences

What it buys

• A bad health-check rollout degrades a service instead of removing it, which is the most likely mass-withdrawal cause.
• The declaration means callers and operators can tell 'the fleet is broken' from 'the health check is broken'.
• The guard's position downstream of evaluation means it survives bugs in evaluation.

What it costs

• Traffic is sent to instances known to be failing, which is worse than a clean failure if those instances corrupt data or hold connections.
• A genuinely broken fleet is kept in rotation, so the user sees errors rather than a fast failure.
• 50% is a cliff, and a service sitting near it will trip and untrip in ways that are confusing to watch.

Choose differently when

For a service where a failing request is unrecoverable for the user or leaves inconsistent state, draining is better than serving and the fraction should be lowered or the guard replaced by a human gate. If the platform could reliably distinguish a broken fleet from a broken health check, it could choose differently in each case and the blunt guard would no longer be needed.

Why it holds up over time

The principle — no single signal may remove a service's capacity — outlives any particular threshold, and applies to every control system with an actuator larger than its sensor's reliability.

What should happen to an instance that keeps changing its mind about whether it is healthy?

Context

Hysteresis reduces oscillation; it does not eliminate it. An instance that fails under load, recovers when traffic is withdrawn, and fails again when traffic returns will oscillate indefinitely, and each transition is a delta pushed to every subscribed client for that service. At sixty thousand subscriptions, a handful of flapping instances across the estate is a measurable fraction of propagation capacity spent on noise, and the churn makes every client's view less stable than either verdict would be. The instance is also, almost always, a bug report: either its health check is wrong or it genuinely cannot sustain its share, and both are facts its owner needs.

Decision

An instance that changes eligibility more than four times in ten minutes is quarantined: held ineligible for an exponentially increasing period, starting at one minute and capped at thirty. Quarantine is surfaced to the owning team as an event with the transition history that produced it, and appears in the service's flap report. Quarantine never applies to a whole service: the fraction guard takes precedence, so quarantining cannot be a path to draining a fleet.

How it is realised on AWS

The evaluator counts transitions per instance over a sliding ten-minute window and writes a quarantine_until timestamp on the eligibility row. Quarantine events are published to the owning team's notification channel and aggregated into a per-service flap rate, which is a reported signal rather than an alarm.

Options weighed

• ChosenEscalating quarantine with owner notification: Stops the churn, and treats the flap as the defect report it usually is rather than hiding it.
• RejectedLonger hysteresis instead of quarantine: Suppresses the symptom across the board and pays detection latency on every instance to fix a problem a few have.
• RejectedQuarantine with manual release only: Safest against premature return and turns every flap into human work at whatever hour it happens.
• Right elsewherePermanent removal after repeated flapping: Right where instances are cheap and immediately replaceable; here it risks removing capacity the service needs for a condition that may be load-induced.

Consequences

What it buys

• Delta churn from noisy instances is bounded, which protects propagation capacity for the changes that matter.
• Client views stay stable, so a flapping instance stops being everyone's problem.
• The owning team receives a specific, evidenced bug report instead of a vague reliability complaint.

What it costs

• A genuinely recovered instance can be held out for up to thirty minutes, which is lost capacity.
• Quarantine can mask a load-induced failure as an instance defect, sending the owner looking in the wrong place.
• Another piece of per-instance state to hold, report and reason about during an incident.

Choose differently when

If capacity were tight enough that holding a recovered instance out for minutes mattered more than view stability, the cap should fall and the escalation flatten. In an estate where instances are replaced in seconds, terminating the flapping instance and letting the orchestrator supply a new one is simpler and better.

Why it holds up over time

Quarantine of an oscillating participant is a standard answer in any system that aggregates noisy votes into a shared decision, and it will outlive this platform's mechanisms.

How does a changed view reach sixty thousand clients inside five seconds?

Context

Three mechanisms are available and they are not close substitutes. Short-TTL DNS is universally consumable, needs no client library, and is bounded by resolver behaviour nobody controls — including resolvers, language runtimes and sidecars that cache beyond the TTL or ignore it entirely. A five-second withdrawal budget cannot be held by a mechanism whose propagation time is a property of other people's software. Streaming subscriptions give sub-second push with exact version tracking, and require a stateful tier holding sixty thousand long-lived connections, which must survive its own deploys and becomes the most delicate routine operation in the platform. Gossip removes the central tier and makes convergence time and message ordering probabilistic, which is difficult to reason about precisely when reasoning matters most.

Decision

Incremental streaming subscriptions are the primary propagation surface, carrying deltas with monotonically increasing view versions. A DNS zone is maintained as a compatibility surface with identical eligibility semantics, for clients that cannot hold a subscription, and its weaker propagation guarantee is stated rather than hidden. A request/response resolution API exists for tooling and target-group synchronisation and is not for request-path use. Gossip is not used.

How it is realised on AWS

An Envoy-compatible xDS tier on EKS holds the subscriptions, sending incremental resource updates with a version per service and falling back to a full view only when a client's version is too old to reconcile. Cloud Map and Route 53 are populated from the same assembled views on a short TTL. Clients re-subscribe with jitter capped at 5% of a population per 10 s window, and the tier drains connections on its own rollout.

Options weighed

• ChosenStreaming primary, DNS as compatibility, API for tooling: The only option that holds a 5 s budget while still serving clients that cannot subscribe.
• RejectedDNS everywhere with aggressive TTLs: No client library to ship, and the propagation guarantee belongs to resolvers the platform does not operate.
• RejectedGossip between data planes: Removes the stateful tier, and replaces an explainable latency with a probabilistic one and no version ordering.
• RejectedLong-poll or periodic full-view fetch: Simple and stateless on the server; at 60,000 clients and a 5 s budget it is a continuous full-view fan-out.
• Right elsewhereGossip within a zone, streaming across zones: Attractive where zone-local convergence dominates and the client fleet is homogeneous enough to trust with a membership protocol.

Consequences

What it buys

• Exact per-client version tracking, which is what makes monotonic application and incremental deltas possible at all.
• Withdrawal propagation is a property of the platform rather than of every client's resolver.
• DNS clients are still served, with a stated weaker guarantee rather than a silent one.

What it costs

• A stateful tier holding 60,000 connections is the platform's most dangerous routine deploy, mitigated by jitter and the static window rather than removed.
• A client library is now required for the primary path, which re-couples correctness to fleet management.
• Two surfaces with different propagation guarantees must be kept semantically identical, which is a continuing test burden.

Choose differently when

If the client fleet were homogeneous and small enough that a full-view fetch per interval were affordable, the stateful tier is unnecessary complexity. If the withdrawal budget could be relaxed to tens of seconds — because local ejection carries the fast path entirely — DNS becomes sufficient and the whole tier can go.

Why it holds up over time

xDS is a current answer to a permanent question. What should outlive it is the requirement for versioned, incremental, monotonic delivery, which any replacement must also provide.

When propagation capacity runs short, which changes get delayed?

Context

A single propagation path with a single latency target has to be sized for the worst case, which here is a regional evacuation generating forty thousand instance state changes a minute against a steady state of two and a half thousand. Sizing for sixteen times steady state is expensive and still arbitrary. But the two kinds of change are not equally urgent, and the asymmetry is large: a withdrawal that arrives late means traffic continues to a failing endpoint and users see errors, while an addition that arrives late means the service runs at slightly less capacity than it has. Treating them identically spends capacity on the cheaper one at precisely the moment the expensive one matters.

Decision

Withdrawal and admission are separately budgeted: 99% of callers see an unplanned withdrawal within 5 s and 99.9% within 15 s, while a newly eligible instance need only reach 99% of callers within 30 s. Under saturation the platform lengthens the batching interval for additions and holds the withdrawal path at its budget, and declares that it has done so. Views are monotonic per client, so a delayed addition can never be applied after a later view that already contains it.

How it is realised on AWS

The view assembler maintains two delta queues per service with different batching windows, and the xDS tier drains the withdrawal queue first. Shed episodes are a reported signal. Version comparability across control-plane replicas comes from a region-wide monotonic counter, so a client that reconnects to a different replica cannot regress.

Options weighed

• ChosenSeparate budgets, shed additions, hold withdrawals: Spends scarce propagation capacity on the change whose lateness users feel.
• RejectedOne propagation path, one budget: Simpler to build and operate, and must be sized for a 16× burst to protect the urgent case.
• RejectedPriority by service tier rather than by change type: Protects important services and lets a tier-1 addition delay a tier-2 withdrawal, which is the wrong axis.
• DeferredPush withdrawals, poll for additions: A cleaner expression of the same asymmetry; it means two mechanisms to keep semantically identical for no gain the queues do not already give.

Consequences

What it buys

• A 16× churn burst becomes a capacity question about additions rather than a correctness question about withdrawals.
• The platform can be sized for steady-state additions and burst withdrawals, which is materially cheaper.
• Monotonic versions mean shedding cannot produce an out-of-order view, only a later one.

What it costs

• During a burst a service runs with less capacity visible to callers than it actually has, for up to the shed interval.
• Two queues with different behaviour is a subtlety that shows up in every propagation-lag investigation.
• The admission budget is deliberately loose, which makes slow start's effectiveness partly dependent on propagation timing.

Choose differently when

If adding capacity late were itself a user-visible failure — an autoscaling-driven service where new capacity is the response to load — the asymmetry narrows and both paths need tight budgets. The decision is also wrong if local ejection were strong enough to make withdrawal propagation uncritical, in which case both budgets could relax.

Why it holds up over time

The asymmetry between removing and adding is a property of what users experience, not of any transport, and it holds for any system that publishes membership.

Who decides how much traffic a newly admitted instance receives in its first minute?

Context

A cold replica has empty caches, unwarmed connection pools and a just-in-time compiler that has not seen production traffic. Giving it an equal share of a service's load the instant it passes readiness is a reliable way to make it fail readiness again, which is the thundering-herd failure in its smallest form. The ramp can live in two places. In the client, it is immediate and proportional to that client's own traffic, and it is only as correct as the least-updated client library in the estate — which, across 450 services and many languages, means it is absent somewhere. In the control plane, published as a rising weight, it is uniform across every client and every heterogeneous data plane, and it is only as smooth as the push interval.

Decision

Slow start is owned by the control plane and expressed as a weight that rises as a function of time in rotation and observed success rate, over an interval of at least 60 s. Clients apply the published weight; they do not compute their own ramp. The same mechanism admits a new deployment version, a recovered instance and a newly started instance, so there is one ramp behaviour rather than three.

How it is realised on AWS

Eligibility rows carry a weight that the evaluator raises on a schedule after admission, and the view assembler publishes weight changes as ordinary deltas. Because admission propagation is budgeted at 30 s, the ramp is published in steps rather than continuously — which is accepted, since the ramp's purpose is to bound the first minute, not to be smooth.

Options weighed

• ChosenControl-plane-published weights: Uniform across a heterogeneous client fleet, which matters more here than proportional smoothness.
• RejectedClient-side ramp from a declared start time: Immediate and traffic-proportional, and correct only where the client library is current — which cannot be assumed.
• DeferredControl plane publishes the policy, clients execute it: The best of both and it still depends on client libraries implementing it; worth revisiting once version spread is under control.
• Right elsewhereNo ramp; rely on readiness to gate admission: Right for stateless services with no warm-up cost, and wrong for anything holding a cache or a pool.

Consequences

What it buys

• Every client ramps identically, including DNS clients and target-group-routed traffic that have no ramp logic at all.
• One mechanism covers deploys, restarts and recoveries, so there is one behaviour to understand and test.
• The ramp is visible in the published view, which makes it debuggable from the caller's side.

What it costs

• The ramp is as coarse as the admission propagation interval, so it is a staircase rather than a curve.
• Weight changes consume propagation capacity for every ramping instance, which at ~900 deploys a day is continuous traffic.
• A service whose warm-up exceeds 60 s is still hurt, and few teams will tune the interval.

Choose differently when

Once client library versions are uniform enough to trust, moving the ramp into the client removes the propagation cost and makes it proportional to actual load, which is strictly better. For a genuinely stateless service the ramp is pure cost and should be configurable to zero.

Why it holds up over time

The decision is really about where to put behaviour when client heterogeneity is the binding constraint, and that constraint recurs in every platform with a client library.

What stops a recovered component from being destroyed by the traffic that was waiting for it?

Context

Recovery is the more dangerous event, and the reason is arithmetic rather than subtle. While something is down, demand does not disappear: clients retry, queues fill, and sixty thousand subscriptions wait to reconnect. The instant the thing recovers, all of that arrives simultaneously against a component that is cold. A recovered availability zone receives the traffic of the two zones that were carrying it. A restarted propagation tier receives every client's re-subscription at once. A recovered instance receives its full share before its first cache is warm. Each of these is a different mechanism arriving at the same failure, which is why one cap is not enough.

Decision

Three independent rate caps are platform defaults rather than tuning. No more than 5% of a client population may reconnect to the control plane in any 10 s window, enforced by jittered randomised backoff. A recovered or new instance reaches full weight over at least 60 s. Traffic shifted back into a recovered zone or region is rate-capped, and where the failover was declared manually the shift-back must be explicitly completed rather than happening automatically.

How it is realised on AWS

The xDS tier rejects re-subscriptions above its admission rate with a retry hint, and client libraries apply full jitter on backoff. Weight ramps come from ADR-12. Zone and region shift-back is a policy-API operation that moves topology preference in capped steps, with the final step requiring confirmation for a manual failover.

Options weighed

• ChosenThree independent caps as defaults: Each mechanism arrives at the same failure by a different route, so each needs its own ceiling.
• RejectedOne global admission control at the control plane: One place to reason about, and it cannot cap traffic shifting between zones, which never touches the control plane.
• RejectedClient-side backoff only: Standard practice and dependent on every client library doing it correctly, which is the assumption this design avoids making.
• RejectedAutomatic shift-back on recovery: Faster return to full capacity and removes the human judgement that the recovery is real.
• DeferredQueue-based admission with explicit draining of the backlog: More precise than rate caps for the reconnection case; the caps are sufficient and much simpler to operate.

Consequences

What it buys

• A control-plane restart does not become a company-wide reconnection storm, which makes routine deploys of the tier survivable.
• A recovered zone is reloaded gradually, so the recovery does not immediately produce a second incident.
• Manual failovers end with a human confirming the return, which is where the judgement belongs.

What it costs

• Full recovery takes longer than it needs to when the recovered component is genuinely healthy.
• A manual shift-back that nobody completes leaves a region drained indefinitely, which needs its own alert.
• Three caps in three places is three sets of parameters, and only one of them is visible from any single view.

Choose differently when

If recovery capacity were provably abundant — a stateless service with no warm-up behind an elastic pool — the caps cost availability for no benefit. Automatic shift-back is correct where failover is itself automatic and cheap to reverse, because then no human was in the loop to confirm anything.

Why it holds up over time

That recovery concentrates demand is a permanent consequence of queueing, and any replacement for these mechanisms will need its own ceilings.

Should there be one global registry, or one per region with federation between them?

Context

A single global registry gives one consistent answer, makes cross-region resolution trivial, and makes the global control plane a dependency of every region's ability to route — which contradicts ADR-01 at the regional scale instead of the request scale. Independent regional registries with asynchronous replication keep each region alive through a partition, and guarantee that the regions will sometimes disagree about which region is healthy. That disagreement arrives precisely when the answer matters most, during a partition, and a system that asks the registry whether a region is reachable is asking the wrong component: the registry's own reachability is the thing in question.

Decision

Each region operates an independent control plane with its own registry authority, able to serve all of its own resolution with no dependency on a global component. Cross-region resolution is served from an asynchronously replicated, read-only aggregate of the peers, and replication loss degrades cross-region resolution only. The signal that declares a region unhealthy is out of band and has no dependency on the registry it describes. Reconciliation after a partition is explicit and logged, never derived from timestamps.

How it is realised on AWS

Three independent regional stacks in us-east-1, eu-west-1 and ap-south-1, each with its own Aurora registry and its own control plane on EKS. Cross-region state replicates asynchronously into read-only aggregates. Route 53 Application Recovery Controller readiness checks and routing controls provide the out-of-band region health and failover signal, deliberately chosen because its data plane is separate from the platform's own.

Options weighed

• ChosenRegional authority with read-only cross-region aggregates and an out-of-band failover signal: A region routes with every peer and every global component unreachable, which is the property that matters.
• RejectedSingle global registry with regional read caches: One consistent answer, and the global plane becomes a dependency of every region's routing.
• RejectedRegional authority with a global read-only aggregate as the single cross-region view: Nearly the chosen design, with one global component whose loss removes all cross-region resolution at once.
• RejectedStrongly consistent multi-region registry: Removes disagreement entirely and makes every registry write pay a cross-region round trip, and a partition stop writes.
• Right elsewhereSingle-region control plane for the whole estate: Right for an estate genuinely in one region; here it makes one region's bad day global.

Consequences

What it buys

• A regional partition is a cross-region resolution degradation rather than a routing outage anywhere.
• The failover decision does not depend on the system whose reachability is in question.
• Each region's control plane can be deployed and drilled independently, which is how the static-serving claim gets tested.

What it costs

• Regions will disagree about reality after a partition, and the reconciliation is explicit work rather than an automatic merge.
• Three stacks to operate, upgrade and keep semantically identical.
• Cross-region resolution is always slightly stale, so a cross-region failover routes on an older view than a local one.

Choose differently when

If the estate's traffic were genuinely global rather than regionally served — every request touching services in two regions — the staleness of a cross-region aggregate becomes a correctness problem and strong consistency starts to earn its cost. A single region of operation makes the whole federation question moot.

Why it holds up over time

Not asking a component about its own reachability is a permanent rule. The specific choice of regions and replication mechanism is not.

Who is allowed to mark an instance unhealthy, and what stops that being an attack?

Context

The discovery plane's central capability is removing endpoints from rotation. Stated plainly, it is an authorised denial-of-service mechanism operating continuously across the entire estate, and most implementations treat its inputs as telemetry rather than as security-relevant assertions. If any workload can report health about any instance, then a single compromised pod can remove a service it merely calls. If an operator can always force an instance ineligible, the break-glass path is a denial-of-service path with a badge. And the one evidence class that cannot be forged about oneself — the outcomes callers observed — can still be forged about somebody else.

Decision

Every registration, health report and resolution request is authenticated with workload identity rather than network position. An instance may report health only about itself; a prober only about the instances assigned to it. A signal that cannot be attributed to a registered identity is discarded, not down-weighted. Marking an instance ineligible by hand is privileged, rate-limited per actor per service, requires a justification, and is audited whether or not it succeeds. A registered address must be proved to belong to the identity registering it. Resolution is authorised per caller-callee pair, so the registry is not a map of the estate for any compromised workload.

How it is realised on AWS

IRSA and instance profiles supply workload identity; all control-plane paths use mutual authentication. Prober assignment is held in the registry, so a prober report outside its assignment is rejected at ingest. Overrides go through the policy API, which checks authorisation and the disruption budget before anything is applied, and writes the attempt to the 5-year immutable audit store. Caller outcome reports are weighted by reporter population rather than by report volume, so one noisy reporter cannot dominate.

Options weighed

• ChosenIdentity-attributed signals, self-only reporting, privileged and audited overrides: Treats the removal capability as what it is, and makes the common attack — report about a neighbour — structurally impossible.
• RejectedTrust signals from any authenticated workload in the mesh: Simple, and gives every workload the ability to deny service to every other.
• RejectedNetwork-position-based trust: Requires no identity plumbing and fails completely the first time anything inside the perimeter is compromised.
• RejectedUnrestricted operator override: Fast under pressure and makes the break-glass credential the most powerful outage tool in the company.
• DeferredSigned health assertions with per-report non-repudiation: Stronger than identity-scoped attribution, and the per-signal cost at 1.2 M results a second does not currently buy enough.

Consequences

What it buys

• A compromised workload cannot remove a service it merely calls, which is the attack this capability otherwise invites.
• Every denial is attributable after the fact, including the ones that were refused.
• Per-pair resolution authorisation means a compromised workload cannot enumerate the estate.

What it costs

• Identity plumbing on every path, including the high-volume passive ingest, where it is the per-sample cost.
• Prober assignment becomes registry state that must be correct, or legitimate probe reports are rejected.
• A refused override under incident pressure is a frustrating experience, and the refusal has to explain itself well to be accepted.

Choose differently when

In a single-tenant estate with no meaningful internal threat model, per-pair resolution authorisation is overhead and a flat mesh identity would do. If incident response ever showed that refused overrides were materially extending outages, the budget check should be made overridable with a second authoriser rather than removed.

Why it holds up over time

Reasoning about a capability by what it can destroy rather than by what it is called is a permanent discipline, and it survives every change of identity technology.

How should workloads that cannot answer a readiness question be represented?

Context

The estate is not uniform and will not become uniform. Alongside twelve orchestrated clusters there are long-lived VM fleets, managed functions behind an invoke path, remaining data-centre services, and third-party endpoints that publish no health signal of any kind. Forcing all of them into one abstraction gives callers a single resolution surface and one mental model, at the price of an abstraction whose health contract is only as rich as its weakest member. Giving the heterogeneous tail a second mechanism keeps the orchestrated path clean and forces every caller to know how its callee is deployed, which is exactly the coupling a discovery plane exists to remove.

Decision

All workloads are services in one registry with one resolution surface. Each service declares a capability profile stating which health signals its runtime can actually supply and which are unavailable, and the platform never reduces a richer service's contract to the weakest profile in the estate. A service with no readiness signal is represented as eligible with health unknown, stated in the view rather than implied, and is governed by passive caller outcomes alone.

How it is realised on AWS

The service row carries a capability_profile; the health contract's required fields are validated against it, so a contract demanding readiness from a third-party endpoint is rejected at authoring time. Views carry the capability per endpoint so a client can tell an unprobed endpoint from a probed healthy one. Non-orchestrated fleets renew leases; orchestrated ones do not need to.

Options weighed

• ChosenOne registry, capability declared per service: Callers get one surface and the weak cases stay honest rather than being hidden behind a uniform contract.
• RejectedOne registry with a synthetic health adapter per runtime: Makes every endpoint look probed, which is a lie the caller cannot detect and will route on.
• RejectedSeparate external-dependency registry with its own semantics: Keeps the orchestrated path pristine and makes every caller know its callee's deployment model.
• RejectedLowest-common-denominator contract for the whole estate: Uniform and simple, and discards the readiness and dependency signals that most of the estate can actually supply.
• Right elsewhereOrchestrator-native discovery plus a side channel for the rest: Right where the non-orchestrated tail is small and shrinking; here it is neither.

Consequences

What it buys

• A caller resolves by name without knowing whether its callee is a pod, a fleet instance or somebody else's endpoint.
• Capability is visible in the view, so 'health unknown' is a fact the client can act on rather than an assumption.
• The orchestrated majority keeps its full contract instead of being levelled down.

What it costs

• Clients must handle an endpoint whose health is unknown, and some will route to it as though it were verified.
• Capability profiles are configuration that can be wrong, and a wrong profile silently weakens or over-promises a contract.
• Four registration paths behind one surface is more platform code than a single-runtime design needs.

Choose differently when

If the non-orchestrated tail shrank to nothing, orchestrator-native discovery becomes sufficient and this registry is a layer with no job. If callers proved unable to handle unknown-health endpoints correctly, a separate registry for them — forcing an explicit decision at the call site — would be the safer design despite the coupling.

Why it holds up over time

Declaring capability rather than assuming it is how any abstraction over heterogeneous providers stays honest, and that outlives the particular runtimes.