# Health Check & Service Discovery

**Solution Architecture v1.0 · Amazon Web Services with an open-source Envoy/xDS data plane · Reliability Architecture · 2026-10 · 21 views · 16 architecture decision records**

When a team chat shows a spinner on send, when a shared document says "reconnecting", when the driver freezes on the map for eight seconds, the user is almost never looking at a crashed service. They are looking at a request routed to a replica that should not have been in rotation — one restarting, one whose connection pool was exhausted, one in a zone already declared unhealthy whose address was still cached in the caller. The opposite failure is less visible and more expensive: a bad health-check config, or a shared database wobbling, and a fleet takes itself out of rotation faster than it can be put back. This package is the internal health and discovery plane that answers two questions continuously for every service-to-service call in an assumed collaboration SaaS: 450 internal services, 60,000 instances, 3 active regions, 12 million internal requests a second at peak, 2,500 instance state changes a minute at steady state and 40,000 during a regional evacuation — on EKS, ECS and EC2 Auto Scaling for registration, Cloud Map and Route 53 for the DNS surface, an Envoy/xDS tier for propagation, Aurora for desired state, DynamoDB for observed state, and ARC for the out-of-band region signal.

The product is not the ability to know which replicas are healthy. It is **the ability to be wrong about it, briefly and boundedly, without that being the same thing as an outage** — because the discovery plane is itself a distributed system that will have a bad day, and on that day every service in the company is downstream of it.

The design rests on one rule: **the client's last-known-good view is authoritative for routing, and the control plane is an advisory service that improves that view and is never in the request path.**

The decisions that carry the design:

- **The data plane routes; the control plane advises.** A view already delivered keeps working, which lets the control plane target 99.95% while resolution as callers experience it is five nines, and turns a registry outage, a propagation-tier deploy and a cross-region partition into one bounded staleness event instead of three outages (ADR-01).
- **Computed state is disposable.** Eligibility and published views are rebuilt from the registry and the sample store, never restored — which is what makes a five-minute RTO a throughput measurement rather than a backup hope (ADR-02).
- **Registration is reconciled, not self-reported.** The orchestrator already knows which pods and tasks exist; asking the workload to announce itself adds a failure mode in which the fleet looks healthy and is quietly short of members (ADR-03).
- **Three evidence classes, passive outcomes primary where traffic allows.** A probe measures a path no caller uses, a self-report can be confidently wrong, and the outcomes real callers saw are both free and the only class a workload cannot forge about itself. Fusing all three is what makes grey failure visible (ADR-04).
- **Unknown is a state.** A system with only healthy and unhealthy drains itself the first time its observation path breaks. Removal on silence requires corroboration from a second class (ADR-05).
- **A degraded shared dependency reduces weight; it does not withdraw.** If every replica checks the same database, honest unreadiness turns a partially functional service into a completely unavailable one (ADR-06).
- **Asymmetric hysteresis over a decaying score, with a hard-failure fast path.** Re-entry is strictly slower than removal, and a closed port never waits for statistics. Damping is bounded at 5 s of the p99 detection budget (ADR-07).
- **The minimum healthy fraction disregards health rather than draining a service.** Below 50% eligible per zone the platform returns the full set and says so, which turns the most common mass-withdrawal cause — a wrong health-check config — into a declared degradation (ADR-08).
- **Flapping is quarantined, and its owner is told.** An oscillating instance destabilises every client's view, and it is almost always a defect report about a health check (ADR-09).
- **Streaming is primary; DNS is compatibility.** A five-second withdrawal budget cannot be held by a mechanism whose propagation latency belongs to other people's resolvers (ADR-10).
- **Withdrawal and admission are budgeted separately, and admission sheds.** Five seconds to remove, thirty to add: adding capacity late is cheaper than adding it wrong (ADR-11).
- **Slow start is published as a weight by the control plane.** Uniform across a heterogeneous client fleet matters more here than being traffic-proportional, because the least-updated client library decides whether a client-side ramp exists at all (ADR-12).
- **Recovery is capped in three places.** Reconnection, traffic share and regional shift-back, because the moment everything is allowed back at once is how a recovered target is killed a second time (ADR-13).
- **Regional authority, with a failover signal that does not depend on the registry.** A region routes with every peer and every global component unreachable, and nothing asks the registry whether the registry can be reached (ADR-14).
- **The power to deny service is a privilege.** An instance may report only about itself, a prober only about its assigned targets, an unattributable signal is discarded, and an override is rate-limited and audited whether or not it succeeds (ADR-15).
- **One registry over heterogeneous runtimes, capability declared per service.** A caller resolves by name whether its callee is a pod, a fleet instance or somebody else's endpoint — and a service with no readiness signal says so rather than levelling the estate to its contract (ADR-16).

The architecture one-pager (including why the design should still hold up in ten years, what a four-week prototype should prove, and the six risks that would change it) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~14 min) and [docs/decision-record.md](docs/decision-record.md) (~60 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 21 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 16 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..c.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..c.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only, no network) |
| `ask.md` | The requirement |

## The twenty-one views

| # | View | What it answers |
|---|---|---|
| 01 | System Context | Three registration sources, one traffic path, and the list of things this plane is not permitted to need |
| 02 | High-Level Architecture | Five stages, and the seam between the fourth and the fifth that the rest of the set is about |
| 03 | Actors and Their Journeys | Four humans and three machines, including the actor whose deepest need is to be told no |
| 04 | Journey — Deploy Without Dropping Traffic | The trough is not the drain: it is the cold replica given a full share the instant it passes readiness |
| 05 | Journey — Chase Errors From One Replica | Every probe green, 4% of requests failing — the journey the passive evidence class exists for |
| 06 | Journey — Evacuate a Zone | A 16× churn burst, a decision made under pressure, and a way back that must be one reversible step |
| 07 | Layered Architecture | Seven layers, six of which are allowed to be unavailable for an hour |
| 08 | Platform Components | Four planes drawn as two boxes, because the control plane fails as one thing and the data plane survives it |
| 09 | Interface Catalogue | Four interfaces in, four out, and exactly one contract in each direction |
| 10 | Data Flow — Signal To Routing Change | One failed request followed to a changed endpoint set, through the attribution check that discards the rest |
| 11 | State Classes | Three classes ordered by what happens if the data is lost; only one has an RPO worth arguing about |
| 12 | Registry Data Model | Eight durable entities, and what is deliberately missing from the view is the point of it |
| 13 | Critical Flow — A Replica Fails | Fourteen messages, and the two that are the reason a user never sees this |
| 14 | Registration To Eligible | Four runtimes with four different amounts of evidence, and one row that is identical for all of them |
| 15 | Eligibility Decision Path | Four kinds of evidence, five verdicts, and two of the verdicts keep the instance in rotation |
| 16 | Deployment Architecture | Three regions of the same shape, none depending on a global component to route |
| 17 | Observability | Six signal families across five stages, reduced to four alarms that map to four distinct actions |
| 18 | Instance Lifecycle | Eight states, and no edge that returns an instance straight to Serving |
| 19 | Trust Zones | Four zones, and the one control the whole design rests on sitting at the ingest boundary |
| 20 | Identity and Authorisation Flow | A privileged action the platform is allowed to refuse, and why the refusal is the control |
| 21 | Failure Classes | Twelve classes, what catches each, and what it costs when the bound fails |

## Rebuilding

```bash
bash scripts/build.sh
```

Node 20+ and nothing else. No draw.io Desktop, no browser, no network. The build assembles
`specs/views.json` and `specs/manifest.json` from the authoring parts, generates the draw.io
sources, fails on any geometry or routing defect, renders the editable and plain SVGs, writes the
HTML pages and the index, injects the one-pager and the decision record, and proves every relative
link in `diagrams/` resolves.

At v1.0 the geometry gate reports 0 errors and 0 warnings across 21 files, the routing gate 0 errors
and 5 clutter warnings across 165 edges, all 346 nodes resolve to an icon with no weak matches, and
the link check passes on 166 relative links. The five warnings are label and edge stacking on the
interface catalogue, the eligibility decision path and the trust-zone view, where several
relationship edges converge on a single node. They were reduced from an initial 30 by cutting the
interface catalogue from six surfaces a side to four, shortening every edge label to the single
constraint that matters, dropping the labels the `sub` already carried, reordering the evidence
column so each row feeds the decision beside it, and moving two long-range edges into the gutter.
Removing the remainder would mean dropping an interface, a verdict path or a zone-crossing edge that
is worth more to the reader than the warning costs.

## A note on the numbers

Every rate, latency, volume, retention and cost figure in this package is a **stated assumption**,
chosen to be defensible and arguable rather than measured. They are stated precisely so that a
reviewer can disagree with one and follow it to the decision that depends on it. `ask.md` marks them
as assumptions section by section; the decision record's evidence note restates the operating context
in one paragraph.