Open Table Format
also called Iceberg, Delta Lake, Hudi
A metadata layer over object storage files that provides atomic commits, schema evolution and time travel, turning a directory of files into a real table.
What these formats fixed is specific. A table represented as "all the Parquet files under this prefix" has no atomicity, so a reader arriving mid-write sees a partial result; no schema evolution beyond hoping every file agrees; no way to delete one record for a data subject request without rewriting a partition by hand; and no way to see what the table looked like yesterday.
The mechanism is a metadata layer that tracks which files constitute the table at each version. Commits become atomic pointer swaps, so readers always see a consistent snapshot. Row-level updates and deletes become expressible. Time travel falls out for free and is genuinely useful for reproducing a model training run or debugging a report that changed.
The strategic significance is decoupling: storage in open formats on object storage, with several engines reading the same tables. That is the credible answer to warehouse lock-in, and the reason this became the main architectural battleground in data platforms.
The operational reality to plan for: metadata grows, small files accumulate, and snapshot expiry and compaction are maintenance jobs someone must own. Formats do not remove the housekeeping, they make it possible.