Your multi-tenant API assigns each customer to one of eight worker pools by hash. One customer's traffic pattern regularly degrades their pool. Improve it.

What the interviewer is testing

Whether you know shuffle sharding and can reason about the combinatorics.

The current state

Hashing to one of eight pools means a customer whose traffic poisons their pool takes down an eighth of the customer base with them. That is a fixed, known, and fairly large blast radius.

The improvement

Assign each customer a random subset of pools rather than one — say two of the eight.

The combinatorics do the work. There are 28 ways to choose two from eight, so two randomly assigned customers share both of their pools only about 4% of the time. A customer whose traffic destroys both of theirs takes down only the small number who drew the identical pair. Everyone else still has at least one healthy pool.

Scale it up and it improves dramatically: 100 workers with 5 each yields combinations in the tens of millions.

The requirement that makes it real

Clients must retry against their alternate assignment. Without that, a customer whose request lands on the degraded pool still fails, and the isolation is theoretical. This is the part that gets overlooked.

The costs

Capacity planning becomes less straightforward — "which pools serve this customer" is a lookup rather than a formula. Per-pool dashboards become less meaningful, because a pool's traffic is now a mix of many customers' partial assignments. And the assignment mapping is state that must be maintained and consistent across the routing layer.

What a strong answer adds

Handling the specific customer separately as well. Shuffle sharding bounds the damage from an unknown future bad actor; a known large or problematic tenant is better placed on dedicated capacity, which additionally isolates their incidents and makes their cost attributable. Most mature multi-tenant platforms do both.

And the observation that the technique applies well below API scale: connection pools, worker queues, rate-limit buckets and cache nodes can all be shuffle sharded, at the cost of a mapping table and retry logic.

Common weak answers

Adding more pools, which reduces blast radius linearly rather than combinatorially. Rate limiting the customer, which helps for volume and not for a pathological request shape.