advanced 3 min answer

A travel-booking platform in Expedia's mould runs analytics on a fixed-capacity appliance at about $1.4M a year with no marginal cost per query. The replacement platform bills per terabyte scanned. Finance approved a business case built from the appliance's CPU utilisation. Sequence the migration so the cost surprise lands before the cutover rather than after it.

migrationpricingbytes scannedparallel runquery log
Show the full answer Hide the answer

Why the business case is already wrong

On an all-you-can-query appliance the marginal cost of a careless query is zero, so the workload has spent years evolving to be wasteful in the exact dimension the new platform charges for. SELECT * over raw tables, dashboards refreshing every five minutes, analysts re-running the same scan, no partition filters because they never mattered. CPU utilisation measures how busy the box was, and it has no conversion to bytes scanned.

The sequence

  1. Harvest 90 days of the query log before touching anything. Distinct queries, their frequency, their tables and columns, and who or what submits them. This artefact drives every later step and it is free.
  2. Price the workload from the log, not from the hardware. Estimate bytes scanned per query from table sizes and the columns referenced. One million queries a month at a mean 4 GB is 4 PB, which at $5 per terabyte is around $20000 a month. Then look at the distribution: the top 2% of queries are usually most of the bytes, so the mean is the wrong number to plan with and the tail is where the bill lives.
  3. Land the data in the layout the pricing model requires - partitioning and clustering chosen from the log's filter columns - before migrating any query. Moving appliance-shaped tables across means paying for the wrong layout on every query, forever.
  4. Replay the top 200 queries against real data on the new platform and measure actual bytes. This converts the estimate into a measurement and finds the handful of queries whose cost is unacceptable while there is still time to materialise an aggregate for them.
  5. Parallel-run with per-team budgets and query cost ceilings switched on from day one, so analysts meet the new economics while the old system is still there to fall back on. A ceiling that rejects a 50 TB scan teaches more in a week than a training session.
  6. Move consumers in cohorts, decommissioning nothing until the last cohort is stable.

Where the numbers diverge

Cost, not data. Expect the replayed bill to differ from the log-based estimate by a factor rather than a percentage, in either direction: pruning on a good layout can remove 90% of the scan, and a dashboard that refreshes on a timer can add a multiple nobody counted. Both surprises are findable in step 4 and unfixable after step 6.

The point of no return

Not the data copy and not the cutover - the hardware or licence contract. Once the renewal is declined, the appliance is gone on a date the vendor chose. Every stage before that is reversible by pointing consumers back. Plan the decommission decision as a dated gate with the replayed cost evidence attached.

How long it really takes

The data copy is days. The layout work, the query replay and the consumer-by-consumer move are months, and they are dominated by finding out what the top consumers actually need. Budget for the parallel run being the longest phase and for running both platforms for a full billing cycle after the last cohort moves.

When this is the wrong answer

If the replacement is bought as committed capacity rather than per-scan, the exercise inverts: the risk becomes over-committing to a baseline, and the number to extract from the log is the concurrency and queue profile at peak hours, not bytes. Price the workload in the unit the contract bills in - that is the only part of this that transfers.