Capacity Borrowing

An organisation running many products on shared infrastructure can meet a sudden, unforecastable demand by reallocating capacity from lower-priority work rather than by provisioning for a peak that may never recur.

This only functions if two things were arranged in advance: a priority order that is decided before the contention, and workloads that can actually be paused without a correctness problem.

Why it matters

The alternative to borrowing is provisioning for the maximum plausible peak, which for a product launch is both unknowable and enormously expensive. Borrowing converts a capacity problem into a prioritisation problem, which is a far better problem to have — provided the prioritisation exists.

Without a defined order, contention resolves arbitrarily, and the arbitrary outcome is frequently that the established revenue-generating product degrades to protect an experiment. That is precisely backwards, and it happens by default.

Implementation patterns

• Explicit priority tiers on every workload, assigned at onboarding rather than during an incident.
• A large pool of genuinely interruptible work as the shock absorber: model training, backfills, batch recomputation, index rebuilds, offline analytics. The size of this pool is the size of the buffer.
• Checkpointing in the displaceable jobs, so that pausing costs progress rather than the whole run.
• Pre-emption that is fast and automatic, since a reallocation that requires a human decision arrives after the spike.
• Rehearsed reallocation, exercised in a game day, because the first attempt during a real launch discovers which jobs cannot in fact be paused.
• A floor per tier, so displacement degrades lower tiers rather than starving them into cascading failure.

Industry example

Threads reached roughly 100 million sign-ups in five days by launching on Instagram's identity, graph and infrastructure. The load was not forecastable to within an order of magnitude, and it was absorbed largely by reallocating capacity across products rather than by pre-provisioning.

What made that possible is structural: a large fleet with a substantial fraction of the workload being offline, batch and training jobs that can yield. An organisation whose entire fleet is serving latency- sensitive traffic has nothing to borrow from, and no amount of policy creates the buffer.

Failure scenarios

• No priority order, so contention is resolved by whichever workload happens to be more aggressive.
• Nothing genuinely interruptible, leaving the buffer theoretical.
• Displaceable jobs without checkpoints, so pre-emption discards hours of work and the team resists it.
• Borrowing across a boundary that should have been isolated, letting the new product's write path saturate the incumbent's critical path — a launch failure becomes an outage of the established product.
• Deferred batch work accumulating, so the borrowing is repaid later as a backlog that must itself be drained.
• Reallocation requiring a human in the loop during a spike measured in minutes.

Trade-offs

Borrowing is far cheaper than provisioning for an unknown peak and it means the shared fleet is a shared failure domain. The displaced work is genuinely delayed, which has costs of its own — stale models, late reports, deferred index rebuilds — and those costs land on teams other than the one that borrowed.

The trade is efficiency and launch agility against isolation and predictability for the workloads that get displaced. It is usually the right trade at large scale and almost never available at small scale, where there is no meaningful pool of interruptible work.

Interview question

"You are launching a product on shared infrastructure and cannot forecast demand within an order of magnitude. Tell me what you need in place beforehand to serve it by reallocation, and tell me the one thing you would refuse to share no matter how much capacity it would save."