Open Table Format
also called Apache Iceberg, Delta Lake, Apache Hudi
A metadata layer over files in object storage that supplies ACID transactions, schema evolution and time travel — the thing that turns a data lake into a lakehouse.
Raw files in object storage cannot support atomic updates, concurrent writers or consistent reads: a job writing a thousand Parquet files is visible partway through, and a failure leaves the dataset in an undefined state.
An open table format adds a transaction log of metadata describing which files constitute the table at each version. A commit is an atomic metadata operation that swaps in a new file list, so readers see either the old version or the new one and never a partial write.
What that unlocks: ACID commits, concurrent writers with optimistic concurrency, row-level upserts and deletes (which raw Parquet cannot do and which GDPR erasure requires), schema evolution without rewriting data, partition evolution, and time travel — querying the table as of a previous version.
The strategic argument is format ownership. Data stays in your object store in an open format that Spark, Trino, Flink, DuckDB, Snowflake and BigQuery can all read, so the compute engine becomes a replaceable choice rather than a decade-long commitment. That is a materially different bargaining position from a proprietary warehouse.