A service runs N supposedly interchangeable replicas, and every request must land on exactly one of them, chosen with information that is stale, local, or both. This guide reconstructs how that choice is actually made at Google, Netflix, Twitter, Uber, Slack, GitHub and Heroku, from their own postmortems, design records and rejected proposals. Afterwards you can pick a selection policy and its guards deliberately, and recognise the four ways the chooser manufactures the imbalance it was installed to remove.
One request, many equal replicas, and a chooser working from signals that lie. Who has solved this in production, and what it costs when the chooser is wrong.
State the problem without naming a product: many copies of the same server, one incoming request, and a component that must assign the request to exactly one copy using whatever it can observe. The copies are never actually equal. Netflix's edge team listed the reasons in 2018: servers are cold right after deployments and autoscaling, they stall during configuration updates and large garbage-collection pauses, and some hardware simply runs permanently slower than the rest. At their scale, over a million requests per second, even a low rate of load-related errors was worth a redesign.
The field looks settled from a distance: round robin, least connections, done. Up close it is one of the most actively contested corners of production engineering. AWS did not offer anything beyond round robin on its Application Load Balancer until November 2019. Envoy shipped latency-aware selection only as a build-time contrib extension. gRPC merged a weighted round robin design in 2023, then argued through two subsetting proposals, closing the first unmerged. And in 2024 Google published a paper about the balancer in front of YouTube whose title is a flat rejection of received wisdom: "Load is not what you should balance." Teams are still changing their answer, which means the decisions are still live.
The single idea that organises everything in this guide: the chooser is a feedback controller, not a dispatcher. Every signal it consumes, a connection count, a latency average, a server's self-reported CPU, is also a path through which its own past decisions, other balancers' decisions, and the failure modes of the fleet flow back into its next decision. Each of the four failure classes in section 4 is one of those feedback paths closing into a loop. The craft is not picking the cleverest scoring function; it is bounding how much the controller is allowed to trust each signal, and for how long.
Scope. This guide covers the selection decision: which live replica gets this request, made by an L7 proxy, a client library or a sidecar. It deliberately excludes consistent hashing for cache affinity, L4 and anycast packet steering (Maglev, Unimog), global traffic steering across regions, overload control once the request has landed, and the separate question of deciding a server is dead, which this site's failure-detection guide already covers. Health checking appears here only where the verdict feeds the pick.
The same four-stage shape appears in Google's GSLB, Netflix's Zuul, Twitter's Finagle, Uber's mesh, Envoy and gRPC. The divergence is in which signal each stage trusts.
Four stages recur in every published system. First a pool view: what the balancer currently believes the fleet is, assembled from discovery and health verdicts. It is a belief, not a fact, and section 4 shows what happens when it drifts. Second a subsetter, present once fleets get large: each client restricts itself to a window of the pool so that connection counts stay affordable. Third the selector proper, which scores candidates and picks one. Fourth a layer of guards that override the score in specific situations: a new host gets a ramp, a suspicious host gets ejected, and a mostly-unhealthy pool triggers a deliberate change of policy.
The convergent default is power of two choices over a local in-flight counter: pick two random candidates, send to the one with fewer outstanding requests. Finagle's documentation calls it the default balancer for all clients; Envoy's least-request policy is "an O(1) algorithm which selects N random available hosts (2 by default)" and cites Mitzenmacher directly, choosing P2C for its "resistance to herding behavior". Netflix's edge rebuilt the same shape with a richer score.
Runs this way at: Twitter (Finagle), Envoy, Netflix
Local counters (in-flight requests) are always fresh but see only this balancer's traffic. Latency averages (peak EWMA) react fast to slow hosts but, per Finagle's own docs, break under long-polling clients. Server-reported load (utilization headers, ORCA reports, probe responses) sees the whole truth but arrives late and can lie, which is why gRFC A58 wraps it in a 10-second blackout, a 3-minute expiry and an error penalty before the scheduler may act on it.
Envoy's slow start scales a new endpoint's weight up over a configured window, because otherwise a new host immediately receives a full proportional share. Outlier ejection removes hosts that keep erroring, covering the blind spot of a selector that finds errors attractive. The panic threshold inverts the whole policy: once fewer than 50% of hosts look available, Envoy stops believing health status entirely rather than concentrating all traffic on the survivors. Finagle ships the same idea as panic mode.
Sources: Envoy slow start, Envoy panic threshold
Two structural divergences matter more than any scoring detail. Where the chooser lives: a dedicated proxy tier (Heroku's router, GitHub's internal balancing layer, ALB), a client library (Finagle, gRPC), or an on-host sidecar (Uber's mesh, Envoy). Client-side picking removes a hop and a point of failure but multiplies the number of independent, non-cooperating controllers; Twitter's SREcon talk is explicit that Finagle's balancing is "non-cooperative, client-side", and the whole aperture design exists to make thousands of selfish choosers produce a fair global outcome. Whether request cost is visible: balancing connections is meaningless once a protocol multiplexes requests over one long-lived connection. The Kubernetes blog's gRPC demo makes this concrete: HTTP/2 keeps a single TCP connection per client, so connection-level balancing parked every request on one pod while the rest idled.
One more recurring component deserves its name said out loud: the load-report aggregator. Uber's control plane collects per-task load and recomputes subset sizes in close to real time; xDS has a load reporting service; Prequal keeps a pool of probe responses per client. Across all three, the aggregated signal feeds weights and subset shapes, never the per-request pick itself, which stays on local state. Nobody published a system where a shared load database sits in the per-request path. That absence is the architecture speaking: a shared signal at pick time is a synchronised signal, and synchronised choosers herd.
Four forks, each with a published team on both sides, and the condition that moves the answer.
Heroku's 2013 crisis is the cautionary tale for the first row of the table below, and it is often retold wrong. Random routing was not a bug; Heroku's FAQ defends it on availability and stateless scaling grounds, saying the team found nothing that "beat the simplicity and robustness of random routing to back-ends that support multiple concurrent connections". The failure was the qualifier: Rails on Thin served one request at a time, so random assignment built queues behind busy dynos while others idled, and the pain concentrated exactly on "Rails apps running on Thin, with six or more dynos, serving 1k requests per minute or more". The press story was marketing ("intelligent routing" stayed on the website after the mid-2010 switch, per The Register); the engineering lesson is that random is a legitimate policy whose precondition is concurrent backends, and the precondition is the part teams forget to check.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Queue awareness at all? | Random routing (Heroku) | Global per-app request queue | Availability and stateless scaling of the router tier | Heroku FAQ, 2013 |
| Default ALB algorithm | Round robin until 2019, then LOR option | LOR as default | AWS added LOR for varied request costs and target churn; RR remains default | AWS, 2019 |
| Scan or sample | P2C (Envoy, Finagle default) | Heap / full scan | Contention and herding resistance; two choices capture the win | Envoy docs, Mitzenmacher, 2001 |
| Selection signal | In-flight + latency (Prequal, Netflix) | CPU as sole signal | Queued work predicts trouble before CPU does | NSDI '24 |
| Self-reported weights | WRR with blackout, expiry, error penalty | Trusting reports immediately and forever | New and stale reports are the two ways the signal lies | gRFC A58, 2023 |
| Subsetting rule in gRPC | Random + rendezvous hashing | Deterministic subsetting (A68 draft) | First proposal closed unmerged; churn and complexity concerns won | PR #383, PR #423 |
| New-host treatment | Slow start window (Envoy), warm-up before trust (A58 blackout) | Full share at join | Cold hosts time out and lose data at full share | Envoy docs |
| Mostly-unhealthy pool | Panic: ignore health or fail fast | Concentrating all traffic on the last healthy hosts | Prevents the survivors being crushed in cascade | Envoy docs |
Four failure classes cover every published incident this hunt found. Each is one of the control loop's feedback paths closing.
Class 1: the chooser rewards failure. A server that fails requests is faster than a server that serves them, so every load metric built on "busy-ness" scores the broken server as the best choice. Google's SRE book states it plainly for least-loaded round robin: a seriously unhealthy task may serve 100% errors, those errors "may have very low latency" because "it's frequently significantly faster to just return an 'I'm unhealthy!' error than to actually process a request", and clients then send the unhealthy task a very large amount of traffic. This is the one class with no named public postmortem in this corpus, and that gap is itself a finding: the mechanism is documented in a book, defended against in gRFC A58's error penalty, Netflix's guardrails and Envoy's outlier ejection, but no organisation has published a dated incident report saying it bit them. Either the defences work, or the incidents get filed under their symptoms. Design as if it will happen: any score derived from completion speed needs an error term before it is safe.
Class 2: herding on shared or stale signals. When many choosers act on the same delayed information, they all draw the same conclusion at once, and the "least loaded" server is buried by the crowd that identified it. Brooker's 2012 and 2024 posts give the mechanism; the C3 paper built its Cassandra replica selector specifically to rank fast servers with growing queues lower, to avoid herd behaviour, and cut p99.9 latency by up to 3x; Envoy's docs justify P2C by its "resistance to herding behavior". The rule the sources converge on: shared signals may shape weights and subsets slowly, but the per-request pick must contain private randomness.
Class 3: the pool view is not reality. The selector can only distribute across the fleet it believes exists. Slack's May 2020 outage is the canonical account: the balancer's belief aged eight hours while autoscaling churned the real fleet underneath it. Class 4: the rebalance is itself an event. Changing the assignment, on a deploy, a scale event or an LB reload, moves load in bulk, synchronously, which is exactly the traffic pattern the steady-state algorithm never sees. GitHub has published three incidents in this class in under two years; Envoy's slow start docs admit the guard does not help "when all the endpoints are relatively new e.g. new deployment in Kubernetes", because a ramp that applies to everyone ramps no one relative to anyone else.
Every figure carries its context and date. Defaults are design decisions someone made for you; the dates tell you how recently anyone reconsidered them.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Connection reduction from subsetting | >99% | Random aperture vs full mesh, aggregate connections (measured) | 2019 | ||
| Hot-instance load under random subsets | ~400 rps | Banding, while other instances sat near idle (measured) | 2019 | ||
| Edge traffic during LB redesign | >1M rps | Netflix | Stated scale motivating the choice-of-2 redesign (claimed by operator) | 2018 | Netflix |
| Probe rate | ~3 per query | Google (YouTube) | Prequal's asynchronous probes, reusable across picks (measured) | 2024 | NSDI '24 |
| Hot/cold RIF threshold | p80 | Google (YouTube) | Replicas above the 80th percentile of recent requests-in-flight are avoided (design value) | 2024 | NSDI '24 |
| Tail-latency gain from herd-aware selection | up to 3x | C3 / Cassandra | p99.9 improvement on EC2 benchmarks (measured, academic) | 2015 | NSDI '15 |
| P2C default sample size | 2 | Envoy | least_request choice_count default; weights equal (design value) | 2026 | Envoy docs |
| Panic threshold | 50% | Envoy, Finagle | Below this availability, health status is disregarded (default in both) | 2026 | Envoy docs |
| Trust blackout for new backends | 10 s | gRPC (A58) | Load reports ignored until reported continuously this long (default) | 2023 | gRFC A58 |
| Weight expiry | 3 min | gRPC (A58) | Stale self-reports stop being used after this (default) | 2023 | gRFC A58 |
| Slow-start floor weight | 10% | Envoy | MinWeightPercent so a warming host still gets picked (default) | 2026 | Envoy docs |
| Queue-time alarm line | 40 ms | Heroku | Average New Relic queue time "indicative of a problem" under random routing | 2013 | Heroku FAQ |
| Stale pool view age at failure | >8 h | Slack | HAProxy instances stuck with stale backend state before the outage (measured) | 2020 | Slack |
| Rebalance blast radius | 1.8% / 15 min | GitHub | Actions runs delayed / average delay, one rebalancing event (measured) | 2026 | GitHub |
| Peak failure rate, retries vs balancer | 75% | GitHub | Search queries failing while retry queues exhausted the LBs (measured) | 2025 | GitHub |
| Imbalance metric | p99 / avg CPU | Uber | The ratio Uber defines as its load-imbalance KPI across a service's tasks | 2022 | Uber |
Netflix's 1M rps is the operator's own scale statement, not an audited measurement. Prequal's production gains ("much higher utilization", lower tails) are reported by its authors without company-external verification; the mechanism numbers (3 probes, p80) are from the paper and slides. Uber published its imbalance metric definition but the before and after ratios sit in figures this session could not extract; treat the metric as usable and the magnitude of Uber's win as unverified. C3's 3x is a benchmark result, not a production figure. Defaults (panic 50%, blackout 10 s) are current as of the dates shown and change with releases.
Every source behind this page, graded. The strongest material is at the tiers ordinary search never surfaces: the postmortems, the design records, and one closed, unmerged proposal.
The definitive pool-view-drift incident: HAProxy state over eight hours stale, autoscaling removing exactly the instances the stale view still trusted, and the one alert that would have caught it silently broken.
A connection rebalancing event inside the internal LB layer skewed traffic across sites and triggered throttling: 1.8% of Actions runs delayed by 15 minutes on average, from housekeeping, not from failure.
Retries first masked flaky LB-to-search connectivity, then exhausted the balancers themselves; up to 75% of search queries failed until the indexing pipeline was slowed.
A provider-side OS upgrade produced "unintended and uneven traffic distribution within the cluster"; the remediation list included closing monitoring gaps for load thresholds.
The platform's official admission that routing behaviour had drifted from its documentation and caused years of unexplained Rails latency; paired with a FAQ that defends random routing and names its precondition.
The recorded argument: a full xDS policy for SRE-book-style deterministic subsetting, drafted, implemented in Go and Java, and closed without merging, with the doc still marked Draft.
What won instead: per-process salted rendezvous hashing over the address list, shipped as the experimental randomsubsetting balancer in grpc-go.
The feature request to port Twitter's d-aperture, with the benefit argued in-thread (fewer connections, even spread, no coordination) and no implementation to date.
The most explicit trust contract in the corpus: weight = qps / (utilization + eps/qps × penalty), a 10 s blackout before a backend's reports count, 3 min expiry after they stop, and an error term so failure cannot masquerade as capacity.
The draft design that carried Google's SRE-book subsetting algorithm into xDS configuration, including the leftover-task refinement; valuable precisely because it was not accepted.
The chapter that documents the error sinkhole ("frequently significantly faster to just return an 'I'm unhealthy!' error than to actually process a request"), deterministic subsetting, and weighted round robin on backend-reported load.
Round robin plus blacklisting was not enough at 1M+ rps; the replacement combines choice-of-2 with the balancer's view first and the server's self-report second, with adaptive guardrails instead of static thresholds.
The three-act subsetting story with production numbers: full mesh unaffordable, random subsets cut connections 99% but banded the load, ring-based determinism restored fairness.
Subset sizes recomputed from aggregated real-time load reports across a mesh of thousands of services in millions of containers; defines the p99/average CPU imbalance metric this guide recommends.
The year-long follow-up: weighting hosts by hardware generation, framed and funded as efficiency work rather than reliability work.
Twelve years apart, the same conclusion from simulation and large-scale deployment: cached global state herds, best-of-k with k of 2 or 3 is nearly as good as perfect information and far more robust to staleness.
The connection-versus-request unit mismatch demonstrated on a live app: one pod took all traffic because HTTP/2 multiplexes onto one long-lived connection.
The early public comparison showing latency-aware policies (least-loaded, peak EWMA) beating the round robin that nginx and HAProxy made the industry default.
The defence of random routing on availability grounds, the 40 ms queue-time alarm line, and the precise description of which applications the policy hurt.
The supermarket-model result the whole field leans on: d=2 gives an exponential improvement over random, d=3 only a constant factor more. The maths that makes sampling two sufficient.
YouTube's balancer: asynchronous reusable probes (~3 per query), hot/cold classification at the 80th percentile of requests-in-flight, lowest latency among the cold. CPU is deliberately not the signal.
Replica selection inside a data store, with ranking built to avoid herding onto fast servers with growing queues; up to 3x p99.9 improvement over Cassandra's stock selection.
A decade after the SRE book published deterministic subsetting, Google wrote up its replacement, designed around reducing connection churn during rollouts.
The talk that names the physics: Poisson arrivals and heavy-tailed service times mean perfect balance is unattainable, and the practical frontier is randomized least-conns, Join-Idle-Queue and careful load interpretation. Cited to the talk page and slide deck; this build environment could not play the video to timestamp claims.
The conference version of the aperture story, explicit that thousands of independent client-side balancers must reach fairness without coordinating. Cited to the talk page and the SREcon APAC slide deck; video not viewable from this build environment.
The least-request policy's two modes, the P2C default with the Mitzenmacher citation in the docs themselves, and the admission that the weighted fallback "will never truly drain" a host.
The cold-host guard and its own failure modes, documented: ineffective when the whole fleet is new, risky at low traffic (starvation, non-gradual weight jumps).
Below 50% availability Envoy stops trusting health entirely, balancing to all hosts or failing outright, with the trade-off between the two modes spelled out.
Unusually honest library documentation: the heap balancer's contention, P2C's degradation to random under low load, Peak EWMA's long-polling blind spot, panic mode's thresholds.
The managed-platform data point: the biggest cloud load balancer offered only round robin until late 2019, and added LOR citing varied request costs and target churn.
Six rungs. The first three fit in an evening each with two toy HTTP servers and any proxy; the last three are the production crossing.
Two identical backends, a proxy assigning uniformly at random, constant request rate. Plot per-backend in-flight counts over ten minutes, then repeat with round robin.
Done when: your plot shows random producing transient 3:1 in-flight skews that round robin does not. Teaches: why "uniform" and "even" are different words, the gap behind Heroku's 2013 trouble.
Make one backend return 500 in 2 ms while the other serves in 120 ms. Switch the proxy to least-connections and watch the traffic share of the broken backend.
Done when: the failing backend receives the clear majority of requests within a minute. Teaches: the SRE book's sinkhole, viscerally; why every busy-ness score needs an error term.
Implement pick-two-compare on in-flight count, then multiply each candidate's score by a penalty from its recent error rate (gRFC A58's eps/qps shape). Re-run rung 2.
Done when: the broken backend's share falls below 5% without any health check marking it down. Teaches: the difference between health checking and selection, and why both exist.
Run Envoy with least_request, slow start (30 s window) and outlier ejection in front of three backends. Kill and cold-start one backend repeatedly while under load.
Done when: a restarting backend's traffic ramps visibly in the Envoy stats and p99 stays flat through restarts. Teaches: guards as config you can read, and what the stats for them look like on a dashboard.
Add a self-reported utilization value to each backend's responses and switch selection to weight on it. Then freeze one backend's reports and make another under-report by half.
Done when: you have reproduced both failure modes and fixed them with a blackout period and an expiry, matching A58's 10 s / 3 min defaults. Teaches: why a signal contract has three clauses, not one.
Fifty client processes, twenty backends, subset size five. Compare random subsets against deterministic ring windows. Track connection count and p99/average load per backend (Uber's metric), then roll all backends and measure connection churn.
Done when: you can state the three-way trade you measured: connections saved, worst-backend load, churn under rollout. Teaches: why Twitter, Google and gRPC each landed on a different subsetting answer.
The queries that actually found this material, in the forms that worked. The vocabulary terms (aperture, banding, requests-in-flight, p99/average) are the keys; each good source taught the next query.
"least connections" load balancer failing server outage postmortem"connection rebalancing" availability report site:github.blogslack outage 2020 haproxy "server state" postmortemheroku "routing performance update" random routing"deterministic aperture" load balancinguber engineering "dynamic subsetting""p99" "average cpu" imbalance load balancingprequal "requests in flight" nsdi youtuberepo:grpc/proposal is:pr is:closed is:unmerged subsettinggrpc proposal "weighted_round_robin" blackout_periodenvoy issue "deterministic aperture""reinventing backend subsetting" googlemitzenmacher "power of two choices" supermarket modelbrooker "best of k" stale load balancing"herd behavior" replica selection tail latencyenvoy "slow start" starvation "low traffic"