Data Modelling & Storage
Columnar formats, table formats and the lakehouse, partitioning, and modelling choices that decide query cost.
5concepts
60flashcards
36minutes of reading
- 01 Columnar Formats and Why They Win What changes when values of the same column sit next to each other on disk, why that makes compression an order of magnitude better, and the workloads where row storage is still correct.
- 02 Dimensional Modelling and the Case for Wide Tables Why star schemas were designed for a storage economics that no longer holds, what they still provide, and how to decide between a normalised model and one wide denormalised table.
- 03 Table Formats and the Lakehouse What a directory of Parquet files cannot do, how a metadata layer adds atomic commits and time travel on top of immutable object storage, and where the abstraction leaks.
- 04 Partitioning, Clustering and File Sizing The three physical layout decisions that determine query cost, why partitioning on a high-cardinality column is the classic disaster, and how to reason about the right file size.
- 05 Predicate Pushdown and Column Statistics How a query engine avoids reading data it can prove is irrelevant, the hierarchy of pruning from partition to page, and why statistics that exist can still be useless.