Data Modelling & Storage
Columnar formats, table formats and the lakehouse, partitioning, and modelling choices that decide query cost.
11concepts
95flashcards
80minutes of reading
- 01 Partitioning, Clustering and File Sizing The three physical layout decisions that determine query cost, why partitioning on a high-cardinality column is the classic disaster, and how to reason about the right file size.
- 02 Predicate Pushdown and Column Statistics How a query engine avoids reading data it can prove is irrelevant, the hierarchy of pruning from partition to page, and why statistics that exist can still be useless.
- 03 Storage Layout for ML Training Reads Why the access pattern of a training loop is the one pattern the lakehouse was not designed for, and how sharded sequential formats trade shuffle quality for two orders of magnitude fewer requests.