Blast Radius

Every system will fail. The design question is not whether, but how many. Blast radius is the answer to "when this component fails or this change is bad, who is affected?" — and it is the single most controllable property in reliability engineering, because it depends on structure rather than on the failure being predicted.

Availability is about frequency; blast radius is about severity. A system with a poor blast radius has incidents that become existential events. A system with a small one has many more visible incidents, each of which affects a small fraction of users, and that is the strictly better position even though it looks worse on an incident count.

Why it matters

Effort spent predicting specific failures has sharply diminishing returns — the next outage will have a cause nobody enumerated. Effort spent bounding the consequences of an unspecified failure pays off regardless of what the cause turns out to be, which makes it the more robust investment.

It also reframes rollout: a change is a failure waiting to be revealed, so the blast radius of a deployment is as important as the blast radius of a component. This is the connection between resilience architecture and delivery practice, and treating them as separate disciplines is why organisations with excellent redundancy still ship global outages.

Implementation patterns

• Cell-based architecture — many independent full stacks, each serving a subset of customers, sharing no state. The primary structural mechanism.
• Shuffle sharding on top of cells, bounding the radius of tenant-caused failures specifically.
• Bulkheads within a service: separate connection pools, thread pools and queues per dependency or per tenant class, so exhaustion in one path does not starve the others.
• Staged rollout with health gates — one instance, one cell, one zone, one region — applied to configuration and data as rigorously as to code, since those change more often and are usually ungated.
• Regional isolation with no shared control plane, and an honest inventory of what is actually global.
• Per-tenant quotas and rate limits, so a single tenant cannot consume a shared resource entirely.
• A named containment boundary for every dependency: for each thing you depend on, state who is affected when it fails. The dependencies with no answer are the ones to fix.

Industry example

The July 2024 CrowdStrike incident is the clearest recent illustration: a content update distributed to millions of endpoints simultaneously, where a malformed file caused kernel-level crashes. The technical defect matters less than the distribution property — there was no meaningful staging between "released" and "on every machine", so the blast radius of one bad file was the entire installed base, across airlines, hospitals, banks and broadcasters at once.

Recovery was compounded by the failure occurring in the kernel: machines would not boot, so remote remediation was impossible and each endpoint required physical intervention. That is blast radius interacting with recovery cost, and it is the pairing that turns an incident into a global event.

The contrasting example is instructive: providers running cell-based architectures have had comparable defects whose impact was a small percentage of customers, because the structure made "everyone at once" impossible.

Failure scenarios

• A global configuration push with no waves, which is the modern equivalent of deploying straight to production everywhere.
• A shared dependency used by every service — service discovery, configuration, identity — whose blast radius is the whole company.
• Multi-tenant infrastructure with no per-tenant limits, so one tenant's spike is everyone's outage.
• A single global database or a single global queue, however well replicated.
• Failure in a layer where recovery requires physical access, which multiplies the cost of a wide radius.
• Observability inside the blast radius, removing the ability to see the failure.
• Cells that share a control plane complex enough to fail on its own, which relocates the problem.

Trade-offs

Reducing blast radius costs efficiency, and sometimes a great deal of it. Cells fragment capacity, so utilisation falls and per-cell overhead multiplies. Staged rollouts mean a change takes hours or days to reach everyone, which is a genuine cost when the change is a security fix and the exposure window matters. Per-tenant limits mean rejecting work the system could have served.

There is also a real operational cost: many small deployments are harder to manage than one large one, and the organisation needs automation mature enough that operating fifty cells is not fifty times the work.

The trade is efficiency and change velocity against severity of the worst day. The deciding question is whether the business can survive the worst case — and where it cannot, the efficiency loss is simply the price of continuing to exist, which reframes the argument from an engineering preference to a business constraint.

Interview question

"Pick any dependency in our system and tell me who is affected when it fails completely. Now do it for our configuration service. If the answer is 'everyone', tell me what you would change first and what it costs us in utilisation."