A ride-hailing and delivery platform operating across eight South-East Asian markets, of the kind Grab operates, has one internal trips_enriched table that 40 teams already read directly. Leadership wants it published as a data product with an owner, a freshness SLO and per-market residency rules. You cannot break the 40 readers and there is no cutover weekend available. Give the sequence.
Show the full answer Hide the answer
The sequence, each step reversible
- Find out who actually reads it. 90 days of warehouse query logs, grouped by identity and by query share. The result is reliably surprising: the 40 teams turn out to be 60 or 70 service accounts and personal identities, a handful typically account for more than 80% of the volume, and roughly a third of the readers touch it once a quarter. No other step in this migration is safe until this list exists, because every later decision is a judgement about who you are willing to break.
- Publish a view, not a new table. Create
trips_product_v1over the existing physical table, exposing only the columns you are prepared to commit to. A view costs nothing to create, nothing to store, and can be dropped. Reserving the right to change the base table is the entire point. - Write the contract before moving anyone. Column semantics, grain, freshness SLO, what happens to a closed partition if a defect is found, the deprecation notice period, and the name of one owner. A data product without a stated change policy is a table with a nicer name.
- Move consumers, measured by query share rather than by tickets closed. Expect the top five readers in a month and a long tail over two quarters. Keep both paths live and compare row counts and a check-sum of a few aggregates between base and view nightly.
- Apply residency on the view before admitting any cross-market reader. Partition by market and attach a row policy keyed to the principal's market entitlement. Doing this at step 5 rather than step 2 matters because the view's numbers now legitimately differ from the base table's for a restricted reader, and you need the comparison in step 4 to have been clean first.
- Revoke direct grants on the base table. This is the point of no return: once revoked, a missed consumer is a broken pipeline rather than a warning. Do it per-reader as each one migrates, never in one change.
- Only now change the physical layout. Repartition, re-cluster, rename columns, split the table. The contract has decoupled you from the readers, which is the thing the migration was actually for.
Where data can diverge, and how you would know
Three places. Residency filtering makes a restricted reader's totals smaller, and if that is not published, an analyst will report a market shortfall that does not exist. A column dropped from the view sends people back to the base table rather than to you, which the query log will show. And a backfill run during step 4 rewrites history on the base table while a consumer holds an exported snapshot from the view; the nightly comparison catches it the following morning, which is the argument for running the comparison at all.
How long it really takes
Six to nine months, and more than half of that is step 4. The engineering is a week. The schedule is set by other teams' sprint capacity, which is why the migration needs the owner named at step 3 to have authority to set a deprecation date and to be contactable when it lands badly.
When this is the wrong answer
When fewer than five teams read the table, skip all of it. Message the five readers, agree the columns in a chat thread, rename nothing, and write the freshness expectation in the table comment. A view, a contract document, an SLO and a deprecation policy for five known consumers is ceremony that costs more than the coordination it replaces. Prefer the full sequence only when you cannot name every reader from memory — roughly where step 1 tells you something you did not already know. Published accounts of data-product programmes since 2020 agree on one point: the engineering is never the schedule.