A service has widely varying request costs — most complete in 10 ms, some take 5 s. Under round robin some instances are overwhelmed while others idle. What do you change?
Show the full answer Hide the answer
Why round robin fails here
It distributes by count, not by cost or by whether a backend is coping. With uniform requests that is fine. With a 500× cost spread, an instance that receives three expensive requests in a row is saturated while its peers are idle — and round robin keeps sending to it.
The change
Least outstanding requests (least connections) is the right default. A saturated instance holds open requests, so it naturally receives fewer. It requires no cost estimation and adapts automatically.
Power of two choices if the balancer is distributed and global state is expensive: pick two backends at random, send to the less loaded. Most of the benefit without the coordination, and what several modern proxies use by default.
Least response time is the most responsive to grey failure — an instance that is up but slow — but can oscillate if undamped, so it needs smoothing.
Avoid hash-based here: it optimises for cache locality and session stickiness, not for balance, and would pin the expensive requests to fixed instances.
The architectural fix behind the balancing fix
Separate the workloads. A 500× cost spread on one pool means fast requests queue behind slow ones regardless of algorithm. Route long-running work to its own pool, or make it asynchronous — accept, return 202 with a status URL, process on a worker fleet.
This is bulkheading, and it is the answer with the longest shelf life.
Health checking, which is usually wrong at the same time
Compute the actual ejection time: interval × unhealthy threshold + timeout. A 30-second interval with a threshold of 3 leaves a dead backend receiving requests for 90 seconds — a figure that belongs in your availability arithmetic and is almost always assumed to be seconds.
Make the check exercise something meaningful; a handler returning 200 unconditionally passes forever while the service is unable to reach its database. And set the healthy threshold higher than the unhealthy one so a flapping instance is not returned to rotation immediately.