Load Balancer Types

Definition

Layer 4 balances connections using IP and port. It cannot see the request, so it cannot route by path or header, and it is fast, cheap and protocol-agnostic.

Layer 7 terminates the connection, parses the request, and can route by path, header, cookie or method. It can retry, rewrite, compress and enforce policy — and it costs CPU and adds latency.

Most estates use both: layer 4 for raw distribution and DDoS absorption at the edge, layer 7 for application routing behind it.

The algorithms, and why the default is often wrong

• Round robin. Even distribution of requests, which is the wrong unit. If requests vary in cost, some instances end up with all the expensive ones.
• Least connections. Sends to the instance with fewest in-flight requests, which approximates sending to the least busy one. Usually a better default than round robin and rarely the default in practice.
• Least response time. Weights by observed latency. Adapts to instances that are degraded but not failing — the case health checks miss entirely.
• Consistent hashing. Routes the same key to the same instance, which matters for cache locality and sticky sessions.
• Power of two choices. Sample two instances at random and pick the less loaded. Nearly as good as globally least-loaded, without needing global state — which is why it is the workhorse of client-side balancing at scale.

Industry example

Netflix's client-side balancing is instructive because it moves the decision to the caller, which knows things a central balancer does not: which availability zone it is in (so it can prefer a local instance and avoid a cross-zone hop and its charges), which instances have recently been slow for its requests, and what its own deadline is.

The cost is that this logic must exist in every client language and cannot be upgraded without redeploying every caller — which is precisely the pressure that later drove the industry toward sidecar proxies, where the same logic sits outside the application and can be upgraded independently.

Failure scenarios

• Health checks that test dependencies, so a downstream slowdown removes healthy instances, reduces capacity, and turns a partial problem into an outage.
• A slow instance still receiving traffic, because it passes health checks while being ten times slower. Latency-aware balancing or outlier detection is the fix; a binary health check cannot see it.
• Sticky sessions used to hide statelessness problems, so a deployment or an instance failure loses user state.
• Connection draining not configured, so every deployment drops in-flight requests.
• Cross-zone balancing enabled without thinking, adding latency and inter-zone transfer charges on every request.

Interview question

"An instance is passing health checks but serving requests ten times slower than its peers. Why is it still receiving traffic and what would you change?"