Bulkheads in Practice

Definition

Partitioning a resource — threads, connections, memory, compute, entire stacks — so that exhaustion in one partition cannot starve another. Named after ship compartments: the point is not to prevent flooding but to confine it.

Why it matters

The most common shape of cascading outage is a shared pool. A service uses one connection pool for all outbound calls; one dependency slows; its calls occupy the pool; the service can no longer serve requests that never touch that dependency.

The failure spreads not because of any logical dependency but because of a shared resource. Two components with no relationship to each other fail together because they drew from the same pool.

Implementation patterns

The pattern appears at every scale, and recognising it as one pattern is the useful part:

Per-dependency connection and thread pools, so a slow downstream exhausts only its own allocation.

Separate pools per workload class — interactive versus batch, so a large report cannot starve user requests.

Separate compute per tenant tier, so a free-tier customer's load cannot degrade paying ones.

Separate clusters across trust or compliance boundaries.

Cell-based architecture at the top of the range, where entire independent stacks each serve a slice of users.

Failure scenarios

Aggregate sizing ignored. Ten pools of 50 connections is 500 per instance; across 20 instances that is 10,000 against a database that accepts 500. Each pool was configured sensibly and the total was not — so the exhaustion moved one layer down rather than being removed. This is the single most common bulkhead mistake.

Sizing at typical rather than maximum scale. A configuration that works at 20 instances fails when autoscaling reaches 60, which is discovered for the first time during a traffic peak.

No wait timeout on acquisition, so a brief slowdown queues requests indefinitely and the service hangs rather than shedding load.

Partitioning something that is not the constraint. Separate thread pools do not help if the actual contention is on a shared database connection or a shared lock.

Industry example

AWS's shuffle sharding is bulkheading with a combinatorial improvement. Rather than assigning each customer to one shard — where a bad tenant takes out everyone in it — each customer gets a random subset of workers. With eight workers and two per customer there are 28 combinations, so two customers share both workers only about 4% of the time. A tenant whose traffic destroys their workers affects only the few who drew the identical pair.

Trade-offs

Utilisation. Partitioned resources idle while another partition is saturated, so total capacity must be higher than a shared pool would need. That is the purchase: containment bought with efficiency.

The counter-position is real — Cloudflare deliberately runs every service on every server precisely to make capacity fungible, accepting that a software failure in one service affects all of them. Which is right depends on whether you are more exposed to uneven demand or to one workload poisoning another.

Interview question

You add separate connection pools per downstream dependency for isolation. Shortly afterwards the database starts refusing connections. What happened?

The candidate should reach aggregate sizing — pools sized individually, total exceeding the database's limit at maximum fleet size. A strong answer sizes backwards from the constraint, proposes a connection proxy where the arithmetic does not work, and notes that pools are usually far larger than optimal anyway.