Error rate has been 1.5% for three days. No alert fired. Customers are complaining. What is happening and what failed?

What the interviewer is testing

Whether you reach gray failure and understand why aggregate alerting misses it.

The likely cause

One instance in the fleet is failing a large fraction of its requests, and the aggregate hides it.

With 20 instances, one failing 30% of its share contributes about 1.5% to the fleet total — below any sensible alert threshold. Meanwhile 1 in 20 users has a substantially broken experience, and retries land on the bad instance again.

Health checks pass because they test something trivial rather than the path that is actually failing. The instance's own view of its health disagrees with its users' — which is the defining property of gray failure.

Typical underlying causes: a failing disk, a degraded network path dropping packets, one exhausted connection pool on that host, a memory leak causing constant garbage collection, or a stale cached credential.

What failed in the detection design

Alerting on the aggregate only. The signal is not the absolute rate; it is that one instance diverges from its peers. Outlier detection on per-instance error rate and latency finds this immediately, and finds it for causes nobody anticipated.

Health checks that do not exercise the real dependency path, returning a constant rather than testing what the service actually needs.

Server-side metrics treated as authoritative when the client's view is the one that matters.

No alerting on burn rate. A sustained 1.5% error rate consumes an error budget steadily, and burn-rate alerting catches exactly this class of slow degradation that never crosses a static threshold.

The response

Eject the outlier rather than diagnosing it live — a fleet member behaving strangely costs more to investigate in place than to replace. Capture its telemetry, terminate it, let the fleet replace it, and analyse afterwards.

What a strong answer adds

That the same reasoning applies one level up: a degraded availability zone or region is the same pattern, and health-check-based failover expecting a binary state does not trigger. Per-zone metric comparison is the equivalent detection.

And the measure worth tracking: what proportion of incidents are detected internally versus reported by customers? Three days of customer-visible errors before anyone knew is a detection failure, and that number says more about observability maturity than any tooling inventory.

Common weak answers

Lowering the alert threshold, which produces noise across the fleet. Restarting everything.