Circuit Breakers in Practice

Definition

A component that monitors calls to a dependency and, after a threshold of failures, stops attempting them — failing fast instead — then periodically probes to see whether the dependency has recovered.

Three states: closed (calls pass), open (calls fail immediately), half-open (limited probes determine whether to close).

Why it matters

Not to reduce errors — an open breaker still returns errors. It exists to stop resource exhaustion in the caller.

The mechanism: a dependency that slows from 50 ms to 3 s holds each caller's thread or connection sixty times longer. By Little's Law, in-flight requests are arrival rate times latency, so occupancy rises sixtyfold and the pool exhausts within seconds. Once exhausted, the caller cannot serve any request — including ones that never touch the slow dependency. A partial failure has become a total one.

The breaker caps that occupancy and, secondarily, gives the struggling dependency room to recover rather than being hammered by retries.

Implementation patterns

Error-rate threshold with a minimum request volume. Three failures out of five is noise; 30% of 200 requests is a signal. A missing volume floor is the most common cause of spurious tripping.

Rolling window evaluation rather than consecutive failures, so a brief blip does not trip it.

Selective failure counting. Connection failures, timeouts and 5xx responses indicate the dependency is unhealthy. A 400 or 404 does not — including them means bad input trips the circuit for everyone.

Half-open with limited probes, so recovery is tested with a few requests rather than by resuming full traffic into a still-fragile dependency.

A fallback. An open circuit fails fast, which helps the caller's resources and not the user unless there is something sensible to return: cached data, a default, a degraded response, a queued retry.

Failure scenarios

Threshold set too close to the dependency's normal error rate, so it trips during ordinary variance and causes the outage it exists to prevent.

Used as a load management tool. If it opens because the dependency is slow-but-working under load, it is doing the wrong job — that is what load shedding and concurrency limits are for.

No fallback, so an open circuit converts slow failure into fast failure with no improvement for the user.

Configuration drift. Every threshold is a guess that decays as traffic grows and the dependency changes. Across hundreds of services nobody keeps them current.

Industry example

Hystrix, Netflix's library, gave a generation of engineers their model of this pattern and was placed in maintenance mode in 2018. Netflix moved towards adaptive concurrency limits, which infer the limit from observed latency — borrowing from TCP congestion control — so the system finds its own capacity and re-finds it whenever conditions change, rather than depending on a human keeping numbers correct.

Trade-offs

Static breakers are easy to understand and explain during an incident, and they require continuous tuning. Adaptive limits self-tune and are harder to reason about, need a clean latency signal, and confuse on dependencies with naturally bimodal latency.

For a small estate with a handful of dependencies, hand-tuned static breakers are entirely reasonable. Past a certain service count, the tuning burden is what decides it.

Interview question

Your circuit breaker opens during normal traffic spikes, causing outages. How do you configure it correctly?

The answer should distinguish what a breaker is for (resource protection against a broken dependency) from what it is not (load management for a slow one), then reach the specific configuration: minimum request volume, rolling window, error-rate rather than count, and counting only failures that indicate dependency health.