advanced 2 min answer

A multi-tenant SaaS platform is degraded for everyone because one enterprise tenant is running ten times its normal workload. What would you change so that this tenant's burst cannot affect the others?

bulkheadsmulti-tenancynoisy-neighboursalesforcewhat-would-you-change
Show the full answer Hide the answer

The diagnosis

Shared resources with no per-tenant accounting. The candidates are almost always the same short list, and usually several at once:

  • Connection pools. One tenant's long-running queries occupy every connection; other tenants queue for a connection before their work even starts.
  • Worker pools. Background jobs run first-in-first-out, so one tenant's bulk import sits ahead of everyone else's interactive work.
  • Database contention. Heavy scans push other tenants' working set out of the buffer cache, so everyone's queries get slower even though nothing is locked.
  • Caches. One tenant's data evicts everyone else's, and hit rates collapse platform-wide.

The common structure: a resource whose consumption is unbounded per tenant, allocated first-come first-served.

What to change

1. Partition the pools, do not just enlarge them. Per-tenant or per-tenant-class limits on connections and concurrent jobs. A tenant that saturates its own allocation queues against itself. This is the essential bulkhead move, and it is more important than any amount of extra capacity — extra capacity is simply a larger pool for one tenant to consume.

2. Fair scheduling instead of FIFO. Weighted fair queueing across tenants for background work, so a tenant with 10,000 queued jobs and one with 3 both make progress. FIFO is not a scheduling policy; it is the absence of one.

3. Quotas with visible, tiered limits. Requests per second, concurrent jobs, query cost units, storage. Enforced, metered, and shown to the customer. A limit the customer cannot see is a limit they will hit by accident and experience as an outage.

4. Shed intelligently at the limit. When a tenant exceeds its allocation, throttle that tenant's lowest-priority work — bulk exports and async imports — before touching interactive requests. Uniform throttling punishes the wrong operations.

5. Cell-based isolation for the largest tenants. Beyond a certain size, sharing is a losing proposition regardless of quota discipline. Place the largest tenants on dedicated cells: full stacks with their own capacity. This caps blast radius structurally rather than by policy, and it also gives commercial optionality — dedicated capacity is something enterprises will pay for.

What not to change

Do not simply raise the limits. It restores service today and guarantees the same incident at the next scale. Absent isolation, capacity is just a bigger shared pool.

Do not isolate everything. Full per-tenant infrastructure for thousands of small tenants is ruinously expensive and operationally unmanageable. The correct shape is tiered: shared with quotas for the many, dedicated cells for the few, and an explicit, measured promotion path between them.

The principle

Multi-tenancy is a bet that tenants' peaks do not coincide. Bulkheads are what you build so that losing the bet costs one tenant's experience instead of everyone's.