advanced 2 min answer

A platform with extreme peaks keeps having total outages caused by bad deployments and poisonous requests. Which pattern addresses this, and what does it cost?

dream11cellsblast-radiusisolationrollout
Show the full answer Hide the answer

The pattern

Cell-based architecture: partition customers into independent cells, each running a full stack — its own application instances, its own data, its own cache, its own queue. A customer belongs to exactly one cell, and a request is routed to it by a thin cell router.

This is redundancy applied to the failure domain rather than to the instance count, and it addresses exactly the failures that replicas do not: a bad deployment, a poisoned configuration, a request that triggers a pathological code path, and overload.

Why replicas do not address those

Three replicas run the same software, deploy through the same pipeline, read the same configuration and share the same dependencies. A software failure is not made less likely by having more copies of it, and the outages that take a whole system down are almost always software, configuration or overload rather than independent hardware failure.

What it costs

  • Operational surface multiplied by cell count. More deployments to track, more monitoring, more places for configuration to drift, more capacity to plan.
  • Cross-cell operations become hard. Anything that must span customers — a global leaderboard, a cross-customer report, an aggregate count — needs an explicit mechanism, and previously trivial queries become distributed.
  • Uneven cell utilisation, since customers are not uniform. Rebalancing a customer between cells is a migration.
  • The cell router is a shared component, and it must be extremely simple and statically stable, or it reintroduces the single point of failure the cells removed.

The property that justifies it

Deployments and configuration changes roll cell by cell, so a bad change affects one cell's customers and is halted before reaching the rest. That converts the most common cause of total outages into a partial, contained one — which for a platform whose peak carries a large share of the business is worth substantial complexity.

The sizing decision

Cells should be small enough that losing one is tolerable and large enough that the operational multiplier is affordable. That is usually a small number of cells rather than many, and the number should be derived from what proportion of customers you can afford to affect — a business decision expressed as an architecture.