advanced 2 min answer

An organisation migrates its warehouse to a new platform. What dominates the effort, and what makes these migrations fail?

warehouse-migrationtransformation-logicconsumersverificationzoomdesign
Show the full answer Hide the answer

What dominates the effort

Not moving the data — reproducing the transformation logic and re-pointing the consumers.

Data movement is a calculable bandwidth exercise. What consumes the schedule:

1. Transformation logic accumulated over years, much of it undocumented, some of it encoding business rules that exist nowhere else. Recovering what it does is the project, and it is the same problem as any legacy assessment.

2. Consumers you cannot enumerate. Dashboards, scheduled exports, spreadsheets connecting directly, applications reading tables, and analysts with saved queries. Most fear of migrating is fear of unknown consumers, which is an information problem solved with query logs.

3. Dialect differences. SQL is not portable in practice — functions, type coercion, null handling, date arithmetic and window semantics differ, and each difference is a potential silent wrong answer rather than an error.

4. Verification. Proving the new platform produces the same numbers requires running both and comparing, which is the step most often compressed and the one that prevents the worst outcome.

What makes them fail

  • Big-bang cutover, concentrating all risk at one moment across every consumer.
  • No verification, so differences are discovered by a business user noticing a number changed.
  • Migrating everything, including reports nobody has opened in two years — measure usage first, because a substantial share is typically unused.
  • No consumer inventory, so the cutover breaks things nobody knew existed.
  • Perpetual coexistence, where both platforms run indefinitely and the business case evaporates.

The approach that works

Incremental, by domain or by consumer group, each with its own verification and cutover. Run both in parallel with automated comparison of the outputs that matter, triaging every discrepancy as a migration defect, an undocumented behaviour to reproduce, or an intended change.

Decommission the old platform per domain as each completes — a domain is not migrated until the old path is switched off, or the organisation operates both permanently.

And measure usage before deciding scope. The cheapest migration is the one where a third of the workload is retired rather than moved.