A team wants to change how search results are ranked on a consumer travel site. The change sits behind a flag and can be reverted in minutes and the guardrail metrics already exist. Which written artefact should they produce to get a decision?
Show the full answer Hide the answer
The deciding property
Cost to reverse, compared with cost to argue. Reverting takes minutes and the metrics exist, so the cheapest route to knowing whether the ranking is better is to run it. The artefact's job is therefore not to argue the design — it is to pre-register what would count as success and when the team will stop.
That means roughly one page: the hypothesis, the primary metric, the guardrails that must not move (conversion, latency, complaint rate), the traffic share and duration, the stopping rule, and who decides at the end. Proposal weight should scale with irreversibility; here it is near zero.
Organisations that run experiments as a default build their written practice around this. Booking.com's experimentation culture, documented in Harvard Business Review's published case study of the company as running on the order of tens of thousands of tests a year, works only because the paperwork per change is small enough that the practice is not rationed. A design document per ranking tweak would cap the organisation's learning rate at its writing rate.
Why the other options fail
- A full design document with alternatives and a migration plan. The right artefact when a decision is hard to undo — a storage engine, a tenancy model, an identifier scheme. Here it spends two weeks of senior time arguing about an outcome that two weeks of traffic would simply report, and it invites reviewers to debate the ranking function instead of the measurement.
- An ADR filed once the change has settled. Decision records are durable and this one deserves one eventually. Filed only afterwards, it produces no decision now, and without a pre-registered metric the record will say "we shipped it and conversion went up", which is the sentence that bakes in a false positive.
- A proposal to the architecture board. Boards should spend their capacity on designs that are expensive to reverse. Putting a flagged ranking change in front of one teaches teams that everything needs permission and adds weeks of queueing to a change that can be undone in minutes.
What would flip the decision
| If this changes | Choose | Because |
|---|---|---|
| The change touches pricing or money movement | full design document | a reversible flag does not reverse a mispriced booking |
| Reaching statistical power takes 6+ weeks | design document plus the experiment plan | you are committing capacity for a quarter and that is a resourcing decision |
| The guardrail metric does not exist yet | a proposal to build the metric first | an experiment without a guardrail is a deploy with extra ceremony |
| The ranking change needs a new data pipeline | design document for the pipeline and a one-pager for the ranking | the irreversible part is the infrastructure |
When this is the wrong answer
When the organisation cannot run an honest experiment — no traffic split, contaminated populations, or no agreed metric owner. Then the one-pager is a fiction and the team needs the heavier document, because argument is genuinely the only evidence available. Equally, in a regulated decision where the regulator expects a rationale written before the change, pre-registration does not substitute for the record.
Common weak answers
- "Write the design document anyway; it is good discipline." Discipline that costs two weeks per reversible change is rationing, and teams respond by not proposing.
- "Skip the document and just ship behind the flag." Without a stopping rule and a guardrail, the flag becomes permanent and nobody can say whether it helped.