Open Table Formats
Iceberg, Delta and Hudi — transactions, snapshots and time travel over object storage.
6 to work through
-
beginner
Your team already writes compressed Parquet files to object storage and queries them with a SQL engine. An engineer proposes moving the same files to Iceberg. The bytes on disk barely change, so what does the table format actually add?
2 min answer -
intermediate Multiple choice
A team has 90 TB of Parquet in Hive-style date directories and wants Iceberg tables with no downtime for the 200 dashboards reading them. Which sequence is right?
3 min answer -
advanced
A team is choosing between Iceberg, Delta and Hudi for a new lakehouse. How would you approach the decision?
1 min answer -
advanced
An Iceberg lakehouse on object storage is healthy. Then the storage service starts serving reads at about four times its usual latency with no errors at all. Nothing is down. Walk through what happens over the next ten minutes and what stops it.
3 min answer -
advanced
An organisation is consolidating a warehouse and a lake onto open table formats. What guarantees do ACID-on-object-storage tables provide, where do small files and compaction bite, and how are streaming and batch writers coordinated?
3 min answer -
advanced Multiple choice
Which requirement most often forces an open table format over plain files in object storage?
2 min answer
5 terms in this topic
Lakehouse Table Format
A metadata layer over object storage that adds atomic commits, snapshots and schema evolution to files that otherwise have none.
conceptMetadata Read Amplification
The number of small sequential metadata requests a table format issues before reading any data, which grows with file history and turns a modest stor…
patternMetadata-Only Migration
Adopting a table format over data files that already exist by generating its metadata in place, so a multi-petabyte conversion becomes a minutes-long…
toolOpen Table Format
A metadata layer over object storage files that provides atomic commits, schema evolution and time travel, turning a directory of files into a real table.
practiceSnapshot Expiry
The scheduled removal of a table's old snapshots and the data files only those snapshots referenced, which is what stops time travel from turning eve…
Neighbouring topics
Data Platform Architecture
General material on designing the analytical data estate end to end.
Medallion Architecture
Bronze, silver and gold layers, and what each layer is allowed to guarantee.
Warehouse, Lake & Lakehouse
Three answers to where analytical data lives, and the workloads that separate them.
Storage Layout & Partitioning
Partition keys, clustering, and the scan the query planner is left able to skip.
File Formats & Compaction
Columnar formats, the small-file problem, and the maintenance nobody schedules.
Ingestion Patterns
Full load, incremental, append-only and merge, and the source system each one suits.
CDC Pipeline Design
Building on a change stream: snapshot plus delta, tombstones, and merge into the target.
Batch Orchestration
DAGs, dependencies, retries, and the difference between a schedule and an orchestration.
Workflow Schedulers
Airflow, Dagster and their kin — where the control plane sits and what it can recover.
Transformation Frameworks
Declarative SQL transformation with tests, lineage and versioned models.
Dimensional Modelling
Facts, dimensions, grain, and the star schema's continued relevance.
Data Vault Modelling
Hubs, links and satellites, and the auditability and load parallelism they buy.
Slowly Changing Dimensions
Overwriting, versioning or timestamping attribute history, and the reporting each enables.
Analytics Cost Control
Scanned bytes, idle warehouses, and the query nobody knew was running hourly.
Workload Isolation
Keeping an analyst's query off the pipeline's compute, and both off the dashboard's.
Data Platform Tenancy
Multiple domains on shared storage and compute, with separable access and cost.
Reverse ETL
Pushing modelled analytical data back into operational systems, and who owns it then.
Data Virtualisation
Querying across sources without moving data, and the performance ceiling that imposes.
Warehouse Migration
Moving off a legacy warehouse with thousands of reports pointed at it.