Scenario-Based Comparison

Comparing designs on "scalability", "maintainability" and "flexibility" produces a discussion rather than a decision. Each participant means something different by each word, and every design scores well on the qualities its advocate values.

Scenarios are comparable in a way qualities are not. "The corpus reaches ten million documents and query volume triples" produces a describable outcome for each option, and the outcomes can be set against each other.

Why it matters

It converts an unresolvable argument about emphasis into an evaluable question about consequence. "More scalable" becomes "handles ten million documents without re-architecting" — which is either true or false.

It also surfaces the scenarios nobody had considered, which is frequently more valuable than the comparison itself.

Implementation patterns

The scenario set should include:

• Growth, at a specific size derived from the business plan rather than from imagination.
• Failure, since designs differ most in how they fail rather than in how they work.
• Change, because most of a system's cost is in changing it.
• Operations — "someone who did not build this debugs a latency problem at 3am" — systematically omitted, and where the difference between a design that is elegant to build and miserable to run becomes visible.
• Exit: what it costs to move away from this choice.

And the discipline:

• Agree the scenario set and its weights before comparing, or the exercise becomes a search for a scenario favouring the preferred option.
• Weight by likelihood, so an option is not chosen for its behaviour in a situation that will not occur.
• Record the outcomes, the decision and the conditions that would change it — the conditions being a monitoring instruction disguised as a decision record.

Industry example

Choosing a search engine illustrates it well: the abstract comparison favours the more powerful option on every axis, while the scenario comparison surfaces that a simpler engine with predictable behaviour and a lower ceiling frequently produces better outcomes than a powerful one tuned badly. The ceiling only matters if you reach it; the tuning burden is paid every day — and only the growth and operations scenarios make that visible.

Failure scenarios

• Feature matrices, where every candidate supports everything and the real decision is unrecorded.
• Scenarios invented after the preference, which is justification rather than analysis.
• No operations scenario, so the design that is pleasant to build wins.
• No exit scenario, so switching cost is discovered later.
• Unweighted scenarios, so an unlikely situation decides it.

Trade-offs

Building a scenario set takes time, and a poorly chosen set can be as misleading as an abstract comparison — particularly if the growth scenario is imagined rather than derived.

The mitigation is to derive scenarios from evidence: the business plan for growth, the incident history for failure, the recent change log for the change scenario. That grounds the exercise and takes little longer than inventing them.

Interview question

"Two designs, one simpler and one more capable. Give me four scenarios you would compare them against, and tell me which one you expect to be decisive."