Metadata-Only Migration

A team has 90 TB of Parquet sitting in date-named directories and 200 dashboards reading it. The proposal on the table is a rewrite: read every file, write it back as an Iceberg table, swap the names. The estimate is eleven days of cluster time and a weekend of downtime nobody will approve.

The rewrite is unnecessary, because the existing Parquet files are already valid data files for the new format. What the directory layout lacks is not a file encoding but a commit protocol and a manifest: a statement of which files belong to the table, with their statistics. Generating that metadata is a write of megabytes, not a rewrite of terabytes. A 90 TB migration becomes minutes per table.

Why it matters

It changes the economics of format adoption from a project to a task, which in turn changes what teams are willing to do. A migration that costs eleven days of compute gets deferred indefinitely and the platform keeps a layout with no atomic commits, no schema enforcement and no time travel. A migration that costs an afternoon gets done.

It also changes the risk profile. A rewrite creates a second copy with a moving cut-off, so for its duration there are two sources of truth and a reconciliation problem. Metadata generation leaves exactly one copy of the data and makes the cutover an atomic pointer change.

Implementation patterns

• Two variants, used in sequence. A snapshot-style migration creates the new table over the same files while leaving the original table intact and writable, giving a rollback window and a parallel-run period. An in-place migration replaces the table. Use the first during cutover, the second once the last writer has moved.
• Inventory every writer per table first, including the ad-hoc scripts. This is the work; the conversion is not.
• Cut writers before readers. A reader on the old path sees stale data and complains. A writer on the old path produces data that is invisible to the new path and produces no complaint at all.
• Expect schema reconciliation. The new format requires one schema per table and will refuse files that do not fit, which surfaces the column added mid-history and the type that changed during an incident three years ago.
• Run one validation query per table across the boundary — a count and a sum by partition on both paths — before moving readers.

Industry example

Every major open table format released since 2017 ships this operation as a first-class command, precisely because full rewrites made adoption uneconomic at the scale where the formats matter. The design that makes it possible is shared: metadata is a pointer structure over immutable data files, so the same bytes can belong to a directory layout and a managed table at once. That property is also what makes time travel and rollback cheap, and it is worth recognising as one decision rather than three features.

Failure scenarios

• A forgotten writer. Files appended to the directory after manifest generation are visible to the old path and invisible to the new one, with no error on either side. Yesterday is quietly missing, and it is found by a business user.
• Migrating in-place with no rollback window, then discovering a reader that depended on directory paths — a partner feed, a legacy script — with no way back except replay.
• Schema drift discovered mid-cutover, halting the migration with half the tables moved and two conventions live.
• Adopting the format and never scheduling maintenance, so within a quarter the accumulated manifests make queries slower than the layout that was replaced.

Trade-offs

When not to use it

Rewrite instead when the layout is what is wrong. If the table is partitioned on a high-cardinality column, or is 8 million small files, adopting a format over that layout preserves the problem and adds a maintenance job to manage it. Do the rewrite on those tables specifically and the metadata swap on the rest, rather than picking one approach for the whole estate.

Below roughly a terabyte the distinction stops mattering — a rewrite is minutes either way, and the rewrite gives you a clean layout for free. And when a writer you do not control appends to the directory on a schedule you cannot change, neither approach fixes the silent-divergence risk: keep the directory as a landing zone and build the managed table downstream of it, accepting one extra copy in exchange for a boundary that fails loudly.

Interview question

Q: You are asked to move 90 TB of Hive-style Parquet onto a managed table format with no downtime for 200 dashboards. Give me the sequence, tell me where the point of no return is, and tell me what will actually consume the elapsed time.

What a strong answer covers: that the data files are already valid and only metadata is written · snapshot-style first for a rollback window, in-place once writers have moved · writers cut before readers, because a stale reader complains and a stray writer does not · the writer inventory and schema drift as the real cost, measured in weeks · a per-table validation query across the boundary · the point of no return being removal of the directory-based writer · and knowing when to rewrite instead, namely when partitioning or file sizes are themselves the problem.

Quick check

Quiz: Why does adopting a table format over 90 TB of existing Parquet take minutes rather than days? Because the format's metadata is a pointer structure over immutable files, so the migration writes manifests rather than moving data.

Flashcard: What is the silent failure during this cutover? A writer still appending into the old directory layout — its files are visible to the old path and invisible to the new one, with no error anywhere.