Table Formats and the Lakehouse

A table used to mean a directory of Parquet files, and a directory of files cannot give you a transaction. Two writers overwrite each other. A reader scanning during a write sees a partial result. Deleting one user's records means rewriting partitions by hand and hoping nothing read them mid-operation. Table formats add a metadata layer that turns that directory into something with database semantics, and it is the change that made the lakehouse a coherent idea rather than a marketing term.

The mechanism

Apache Iceberg, Delta Lake and Apache Hudi differ in detail and share a design. Data files remain immutable Parquet or ORC. Alongside them sits metadata listing exactly which files constitute the table at a given version, along with per-file statistics.

A write produces new data files and then a new metadata version referencing them. The commit is the atomic swap of a single pointer to the current metadata, so readers see either the old table or the new one and never a mix. Iceberg swaps a pointer to a metadata file; Delta appends a JSON entry to a transaction log with periodic checkpoints. Both reduce table-level atomicity to a single-object atomic operation, which object stores can provide.

Everything else follows. Time travel works because old metadata versions still reference their files, so querying the table as of a timestamp is reading an old manifest. Schema evolution works because columns are tracked by stable identifier rather than by position, so renaming or reordering does not require rewriting data. Row-level deletes and updates work through either copy-on-write, rewriting affected files, or merge-on-read, writing delete vectors that readers apply on the fly.

Why file-level statistics beat directory partitioning

Hive-style partitioning encodes partition values in directory paths, and query engines prune by matching path patterns. It works and it is fragile: a query filtering on event_date cannot use a table partitioned by year/month/day unless the engine understands the derivation, so users must filter on the physical layout rather than on the logical column, and changing the partitioning means rewriting the table.

Table formats keep partition values in metadata and track which files hold which ranges of every column. Pruning becomes a metadata query, filters on any column can prune, and Iceberg's hidden partitioning stores the transformation so users filter on the logical column and pruning still happens. Partition evolution becomes a metadata change that leaves existing files alone.

When it breaks

Small files and metadata bloat. Streaming writes produce many small files and a metadata entry per file. Query planning slows as the manifest grows, and eventually planning dominates execution. Compaction and snapshot expiry are mandatory background operations, and a table without them degrades predictably over months.

Concurrent writers still conflict. Optimistic concurrency means a writer whose files were invalidated by a competing commit must retry. Two jobs updating overlapping partitions can livelock under load, and the resolution is partitioning writes by ownership rather than tuning the retry policy.

Merge-on-read defers cost, it does not remove it. Delete vectors make writes fast and every subsequent read slower, since readers apply them. Without compaction the read penalty grows monotonically, so the choice between copy-on-write and merge-on-read is a decision about read-write ratio that has to be revisited as workloads change.

Deletion for compliance is subtler than a DELETE. Removing a row from the current snapshot leaves it in older snapshots that time travel can still read. Genuine erasure requires expiring those snapshots and rewriting the underlying files, which is a separate operation people discover after their first data subject request.

Format choice is a genuine lock-in decision. Engine support differs, catalogue integration differs, and migrating a petabyte-scale table between formats is a rewrite. Interoperability layers exist and translate metadata rather than making the formats equivalent.