How would you determine whether a migration completed six months ago was worth doing?

What is being tested

Whether you close the loop, and whether you understand the attribution problem.

What should have been done at the start

The expected outcome and its measure stated in the design document, before the work began:

"This migration should reduce p99 checkout latency from 1.2 s to under 500 ms and reduce infrastructure cost per order by 20%. We will measure at 30 and 90 days. If latency has not improved by at least 40%, the migration has not achieved its purpose and we will reassess."

Written beforehand it is a prediction, and predictions are how anyone learns. Written afterwards the outcome is chosen to fit whatever happened, which teaches nothing.

Stating the failure criterion is what distinguishes a measurable claim from a hope.

What to measure now

1. The intended outcome, in the terms originally stated. If they were not stated, reconstruct them honestly from the original justification rather than from what turned out well.

2. The counterfactual, which is the hard part. Did the metric move because of this, or because traffic changed, or because three other things shipped that quarter? Without a holdout or a staged rollout, attribution is a story rather than a measurement. Say so explicitly rather than claiming certainty.

3. Unintended effects. Cost elsewhere, latency elsewhere, incident rate, developer experience. Improvements in one dimension frequently move cost into another, and only looking at the intended metric hides that.

4. Whether the assumptions held. The traffic forecast, the cost per unit, the capacity model. When an assumption broke, that is the most valuable finding — it improves every future estimate.

5. The actual cost. Effort, elapsed time, incidents caused during the transition. Compare with what was estimated.

The honest reporting obligation

Report the disappointing ones. An organisation that only records successes learns nothing, and everyone knows the records are unreliable, which destroys the credibility of the practice entirely.

"We predicted a 60% latency improvement and achieved 25%, because assumption X did not hold" is far more valuable than a success story, and it is what makes the next prediction better.

Making it happen at all

• A review scheduled at design time, in the calendar, with a named owner. It will not happen otherwise — the team has moved on and measuring risks discovering the expensive thing did not help.
• A short written record. A paragraph, not a report.
• Results fed into the next decision. "Last time we assumed X and observed Y" is the most valuable input to any estimate an organisation has.

What a strong answer adds

That "was it worth it" and "would we do it again" are different questions with different answers. A migration may have delivered less than hoped and still have been the right call given what was known — and separating the decision quality from the outcome is what allows an organisation to take reasonable risks without punishing the ones that did not pay off.