Sensitivity Analysis

Every architectural comparison rests on assumptions: expected traffic, growth rate, read/write ratio, data volume, team size, the cost of a dependency. Most of them are estimates and some are guesses.

Sensitivity analysis asks a single question of each: if this were wrong by a factor of two, would the recommendation change?

Most assumptions turn out not to matter — the answer is the same across a wide range. One or two dominate, and those are the ones worth measuring, monitoring and designing around.

Why it is the most valuable output of a comparison

A trade-off analysis produces a recommendation, which is useful once. Sensitivity analysis produces a list of what to measure and what to keep changeable, which is useful continuously.

It also converts a debate about a number into a debate about a range, which is far more productive: two people disagreeing about expected traffic can usually agree on a range, and if the recommendation is stable across it, the disagreement was irrelevant.

Implementation patterns

• Vary one assumption at a time across a plausible range, and note where the recommendation flips.
• Report the flip points, not just the recommendation. "This design is right below roughly N requests per second; above that, the other option wins" is far more useful than a verdict.
• Instrument the dominant assumptions so their movement is visible — which is the same practice as an assumption register, applied to decisions rather than to capacity.
• Design the dominant assumption to be cheap to change. If a design's cost is dominated by one uncertain quantity, build so that quantity can be adjusted without a rewrite — a routing layer, a configuration value, an interface.
• State the assumptions you did not test, so a reader can challenge them.

Industry example

Design-stage cost models illustrate it best. The most common failure is precision without accuracy — a number to four significant figures resting on an unexamined traffic assumption, where the assumption is the estimate and the arithmetic is decoration.

Sensitivity analysis exposes that immediately: if doubling the traffic assumption changes the cost by an order of magnitude and changes which architecture is cheaper, then measuring traffic early is worth more than refining the model. If it changes nothing, the model is done.

The same applies to capacity decisions under genuine uncertainty, where the correct response is to commit against a distribution with the asymmetric cost of error made explicit, rather than against a point forecast — and to choose the option that is least bad across the range rather than best at the midpoint.

Failure scenarios

• Presenting a single scenario with no range, inviting a debate about the number rather than the decision.
• Testing only favourable variations, which is rationalisation.
• Identifying the dominant assumption and not instrumenting it, so its movement is discovered later.
• Treating the analysis as a one-off, when the dominant assumption is exactly what should be monitored for the life of the decision.

Trade-offs

It takes additional effort in a review that teams would rather spend elsewhere, and it produces a more complicated answer than a single recommendation — some audiences find "it depends on this number" less satisfying than a verdict.

For a reversible decision it is not worth doing; the decision itself is cheaper than the analysis. For an irreversible one whose cost scales super-linearly, it is one of the highest-return hours available.

Interview question

"You have modelled two architectures and one is 30% cheaper. Before recommending it, what would you check — and what would you do if the answer depended entirely on a traffic figure nobody can confirm?"