Degradation Blindness

Resilience design overwhelmingly assumes a binary: a dependency is available or it is not. Health checks return up or down. Circuit breakers trip on errors. Failover triggers on unreachability. Runbooks describe what to do when something is down.

The state that causes most serious incidents is neither. A dependency that degrades to ten or a hundred times its normal latency is returning correct answers, passing every health check, tripping no breaker, and saturating every caller in the system.

Why it matters

The consequences compound in a specific and predictable order:

• Callers' threads and connections are held, so by Little's law the concurrency required for the same throughput rises proportionally with latency. A 20× latency increase requires 20× the concurrency, which no pool has.
• The caller's own latency rises, and it now appears degraded to its callers. Saturation propagates upward as slowness rather than as errors, which is why the alert usually fires on the wrong service.
• Retries fire, tripling load on a dependency that is already saturated — the standard, normally-correct retry behaviour doing exactly what it was designed to do, at the worst possible moment.
• Failover does not trigger, because nothing is unreachable.
• Fallbacks do not engage, because no error was returned.

Every mechanism designed for resilience is inert, and the mechanism designed for transient errors is actively harmful.

Implementation patterns

• Circuit breakers keyed on latency and concurrency, not only on error rate — an error-only breaker never opens in this scenario.
• Deadline propagation, so a service with 40 ms of remaining budget refuses to start a 2-second call. This converts "slow" into "fast failure" at every layer and eliminates the work whose callers have already given up.
• Retry budgets capping retries as a fraction of total requests, so widespread degradation exhausts the budget and retries stop — converting an amplifier into a ceiling.
• Adaptive concurrency limits that shrink the in-flight cap as latency rises, shedding rather than queueing.
• Bounded pools that fail fast when exhausted, since queueing in front of a saturated dependency converts a capacity problem into a latency problem for everyone.
• Health checks that assess latency, not merely responsiveness.
• Concurrency and queue depth monitored per dependency, which is the signal that actually identifies the saturated component.
• Latency injection in chaos experiments, deliberately, since most fault-injection tooling kills things and the dangerous mode is slowness.

Industry example

The Roblox 2021 outage is the clearest large-scale illustration: a service-discovery and configuration cluster degraded under lock contention and served requests slowly rather than failing. Every dependent service remained "healthy", retries added load to the contended cluster, and the degradation propagated across the entire platform for approximately 73 hours.

The pattern recurs constantly at smaller scale. Google's SRE literature is explicit that timeouts without deadline propagation are among the most common contributors to cascading failure, precisely because they allow each layer to independently authorise work that is already pointless.

Failure scenarios

• Health checks that only verify a process responds, reporting healthy throughout a severe degradation.
• Error-rate-only circuit breakers, which never open.
• Per-call retry counts rather than budgets, so the worse things get the more retries occur.
• Unbounded connection pools or queues, converting saturation into unbounded latency.
• Alerts on error rate alone, silent while latency climbs.
• Failover keyed on unreachability, which never triggers.
• Chaos experiments that only terminate instances, never testing the mode that actually happens.
• Capacity planning assuming normal service times, with no model for the concurrency required when latency multiplies.

Trade-offs

Latency-aware protection fails requests that might have succeeded. A breaker that opens on elevated latency sheds work the system could conceivably have absorbed, and a tight deadline rejects requests during ordinary variability. The tuning is genuinely difficult: too sensitive and availability suffers during normal fluctuation, too lax and the mechanism does not engage when it matters.

Deadline propagation additionally requires end-to-end adoption — a single service that ignores the deadline reintroduces the whole problem — which makes it a platform concern delivered through shared libraries or a mesh rather than a per-team one.

The trade is a higher failure rate under load in exchange for bounded latency and the elimination of work-nobody-wants. At scale that is almost always correct, because unbounded latency is itself a failure, merely one that is harder to see — and the alternative is discovering it during the incident, when every protective mechanism turns out to be inert.

Interview question

"A dependency slows from 50 ms to 2 seconds. Walk me through what happens in our system over the next ten minutes, tell me which of our existing protections engage, and then tell me what you would add — in priority order — and why raising our timeout would make it worse."