Bulkheads & Isolation
Partitioning resources so one dependency cannot starve the others.
5 to work through
-
advanced
A multi-tenant SaaS platform is degraded for everyone because one enterprise tenant is running ten times its normal workload. What would you change so that this tenant's burst cannot affect the others?
2 min answer -
advanced
A service calls four dependencies. How do you size its thread pools, and why is the total often smaller than people expect?
2 min answer -
advanced
A trading platform receives a burst of market-data updates while thousands of customers submit orders. How should market-data distribution, order execution, risk checks and user-facing APIs be separated so that non-critical workloads cannot interfere with trading?
2 min answer -
advanced
You add separate connection pools per downstream dependency for isolation. Shortly afterwards the database starts refusing connections. What happened?
2 min answer -
advanced
You are asked to make a multi-tenant SaaS resilient to "any single failure". You propose cells. What must you find before that claim is true?
2 min answer
6 terms in this topic
Bulkhead
Partitioning resources so that exhaustion in one area cannot consume the capacity another area depends on.
patternBulkheads in Practice
Partitioning resources so one dependency's failure cannot consume what another needs, and the aggregate-sizing mistake that moves the exhaustion one …
patternCell Isolation
Bulkheading at the level of a complete system copy, so that any failure — including ones nobody predicted — is contained to the customers assigned to…
patternPer-Dependency Concurrency Limit
A hard cap on in-flight calls to a specific downstream, which bounds the damage a slow dependency can do regardless of how slow it becomes - more rel…
patternShuffle Sharding
Assigning each tenant a random subset of the available capacity rather than a single shard, so that any two tenants rarely share their entire subset …
patternThread Pool Isolation
Giving each downstream dependency its own pool of threads or permits, so one slow dependency cannot consume the capacity needed to serve everything else.
Neighbouring topics
Distributed Systems
General material on partial failure, coordination and distributed reasoning.
CAP & PACELC
What you must give up during a partition, and the latency choice the rest of the time.
Consistency Models
Linearizable, sequential, causal, eventual, and the session guarantees between them.
Idempotency
Making an operation safe to repeat, because a client that times out cannot know.
Retries & Backoff
Exponential backoff, jitter, retry budgets, and how retries become the outage.
Timeouts & Deadlines
Per-hop timeouts that do not compose, and the deadline budget that replaces them.
Circuit Breakers
Failing fast on a broken dependency, and what you fail fast to.
Backpressure & Flow Control
Telling callers to slow down instead of buffering into congestion collapse.
Load Shedding
Rejecting some work deliberately so the rest can be served correctly.
Leader Election
Agreeing who is in charge, and fencing the one who no longer is.
Consensus Protocols
Raft, Paxos and quorums — what they guarantee and what they cost.
Distributed Locking
Mutual exclusion across machines, and why it is harder than it looks.
Distributed Transactions
Two-phase commit, its blocking failure mode, and when it is still reasonable.
Sagas & Compensation
Replacing atomicity with semantic undo, and ordering the irreversible steps last.
Service Discovery
Finding a healthy address for something whose instances are ephemeral.
Messaging & Queues
Decoupling producer from consumer, and the semantics that come with it.
Event Streaming
Retained ordered logs, consumer offsets, partitions and replay.
Clocks & Ordering
Why wall clocks lie, and how logical clocks and versions restore order.
Failure Modes
Slow rather than down, partial, grey, and failing while reporting success.