A team is choosing between Iceberg, Delta and Hudi for a new lakehouse. How would you approach the decision?
Show the full answer Hide the answer
The formats are more alike than the debate suggests
All three provide the same core: atomic commits over object storage, snapshots and time travel, schema evolution, and metadata that lets a query planner skip files. For most workloads any of them works.
So decide on the surrounding facts rather than on feature comparison.
The three that actually decide it
Which engines must read it. This is usually decisive. Support varies by engine and by version, and a format your query engine, your streaming framework and your BI tool all read natively is worth more than a marginal feature advantage. Check the versions you actually run, not the documentation's claims.
Which your platform provider optimises for. A managed platform will have a first-class format and a supported-but-secondary one. The difference shows up in performance and in support conversations, not in the feature matrix.
Write pattern. Heavy upsert and streaming ingest workloads exercise the formats differently, and this is where the genuine differences are. If that describes your workload, benchmark it rather than reading about it.
The properties to insist on regardless
Catalogue independence — the metadata should not be locked to one vendor's catalogue, or engine independence is theoretical.
A maintenance plan — compaction, snapshot expiry, orphan file cleanup. These are scheduled jobs someone owns, and a table without them degrades in cost and performance until somebody notices.
An exit test. Read the table from a second engine before committing. If that is awkward now, it will be a migration later.
What to tell the team
The format choice matters less than the layout and maintenance decisions they will make afterwards — partitioning, file sizing, compaction cadence. Those determine cost and performance by a wider margin than the format does, and they are where the attention should go.