Health Check & Service Discovery  ·  View 05 of 21  ·  People and journeys

Journey — Chase Errors From One Replica

Every probe is green and 4% of requests are failing. This is the journey the passive signal class exists for.

Editable source SVG draw.io All views
SRE on call paged at 02:40 Goal — Find out whether this is one replica, one zone, or my own health check — in under two minutes Trigger — A caller's error rate rises 4%; every probe is green Done when — The bad replica is out of rotation and the reason is on the record 1 · Page error rate 2 · Localise ◆ moment of truth 3 · Explain 4 · Act ◆ moment of truth 5 · Close What they do Open the service page Compare per-endpoint rates Read the evidence trail Confirm or override File the flap report What the plane shows Eligible fraction 100% One endpoint, 40% fail Probe green, passive red Weight already cut Transition + cause How it feels In control Hunting Blind Where it hurts Grey failure probes clean Override is a denial of service What answers it Per-endpoint outcomes Passive signal is primary Inputs exposed per decision Override audited, rate-limited Quarantine, not a guess Journey — Chase Errors Coming From One Replica v 1.0 · owner Reliability Architecture

What answers the trough

  • Per-endpoint outcomes reported by callers, so grey failure is visible without the prober being able to see it.
  • Every eligibility decision exposes the inputs that produced it, which is what turns a dashboard into an explanation.
  • The override exists, and it is rate-limited and audited, because it is a denial-of-service path with a badge.

Assumptions

  • Grey failure (brownout, exhausted pool, GC pause) is assumed to be the most common real failure, not process death.
  • Detection within 3 s p50 and 10 s p99 of the first failed request.

Risks

  • Passive signal is unavailable for a low-traffic service, which is exactly where probing remains the only evidence.
◀  Journey — Deploy Without Dropping Traffic All views Journey — Evacuate a Zone, and Bring It Back  ▶
Solution Architecture v1.0 · Amazon Web Services with an open-source Envoy/xDS data plane · Reliability Architecture · 2026-10 · 21 views · 16 architecture decision records05-journey-chase-errors-from-one-replica