Warehouse Migration
Moving off a legacy warehouse with thousands of reports pointed at it.
5 to work through
-
advanced
A retail company is migrating its warehouse to a new platform. What makes these migrations fail, and what sequence reduces the risk?
2 min answer -
advanced
An organisation is migrating from a legacy data warehouse to a modern platform. How should it be sequenced, and why do these migrations so often end with both systems running permanently?
2 min answer -
advanced
An organisation migrates its warehouse to a new platform. What dominates the effort, and what makes these migrations fail?
2 min answer -
advanced
During a warehouse migration parallel run, 40 reports produce different numbers on the new platform. How do you proceed?
2 min answer -
advanced
Your analytics platform ingests change data from an operational estate that was split into 480 logical shards in 2021 and then spread across a larger number of physical database hosts in 2023, the arrangement Notion has described publicly. Your connectors, offsets and target tables are all configured per host. What does that physical move do to the downstream pipeline, and what design turns the next one into a configuration change rather than a re-ingest?
3 min answer
3 terms in this topic
Legitimate Divergence
The class of differences between an old and a new analytical platform that are correct on both sides, which a parallel run must classify and budget f…
practiceReport Migration Inventory
The enumerated list of every report, extract and downstream consumer of the legacy warehouse, with usage evidence, which is what makes the migration …
practiceWarehouse Migration
Moving analytics from one platform to another, where the difficulty is almost never the data and almost always the accumulated logic and consumers.
Neighbouring topics
Data Platform Architecture
General material on designing the analytical data estate end to end.
Medallion Architecture
Bronze, silver and gold layers, and what each layer is allowed to guarantee.
Open Table Formats
Iceberg, Delta and Hudi — transactions, snapshots and time travel over object storage.
Warehouse, Lake & Lakehouse
Three answers to where analytical data lives, and the workloads that separate them.
Storage Layout & Partitioning
Partition keys, clustering, and the scan the query planner is left able to skip.
File Formats & Compaction
Columnar formats, the small-file problem, and the maintenance nobody schedules.
Ingestion Patterns
Full load, incremental, append-only and merge, and the source system each one suits.
CDC Pipeline Design
Building on a change stream: snapshot plus delta, tombstones, and merge into the target.
Batch Orchestration
DAGs, dependencies, retries, and the difference between a schedule and an orchestration.
Workflow Schedulers
Airflow, Dagster and their kin — where the control plane sits and what it can recover.
Transformation Frameworks
Declarative SQL transformation with tests, lineage and versioned models.
Dimensional Modelling
Facts, dimensions, grain, and the star schema's continued relevance.
Data Vault Modelling
Hubs, links and satellites, and the auditability and load parallelism they buy.
Slowly Changing Dimensions
Overwriting, versioning or timestamping attribute history, and the reporting each enables.
Analytics Cost Control
Scanned bytes, idle warehouses, and the query nobody knew was running hourly.
Workload Isolation
Keeping an analyst's query off the pipeline's compute, and both off the dashboard's.
Data Platform Tenancy
Multiple domains on shared storage and compute, with separable access and cost.
Reverse ETL
Pushing modelled analytical data back into operational systems, and who owns it then.
Data Virtualisation
Querying across sources without moving data, and the performance ceiling that imposes.