Data Modelling & Storage
Columnar formats, table formats and the lakehouse, partitioning, and modelling choices that decide query cost.
11concepts
95flashcards
80minutes of reading
- 01 Arrow and Zero-Copy Columnar Interchange The difference between a storage format and a memory format, why Arrow eliminated a conversion step that cost more than the query, and where copies still happen despite the name.
- 02 Columnar Formats and Why They Win What changes when values of the same column sit next to each other on disk, why that makes compression an order of magnitude better, and the workloads where row storage is still correct.
- 03 Compaction and the Small File Problem Why streaming ingestion destroys query performance on a table that is otherwise perfectly healthy, what compaction actually costs, and the write-amplification trap in naive compaction policies.
- 04 Copy-on-Write versus Merge-on-Read The two ways to change one row inside an immutable file, why the choice is really a decision about where to spend amplification, and what deletion vectors changed.
- 05 Dimensional Modelling and the Case for Wide Tables Why star schemas were designed for a storage economics that no longer holds, what they still provide, and how to decide between a normalised model and one wide denormalised table.
- 06 Object Storage Semantics for Analytics Why a lakehouse built on S3 behaves nothing like one built on a filesystem, and the four properties of object storage that decide table layout, commit protocols and query cost.
- 07 Slowly Changing Dimensions and History How SCD types 1, 2, 3 and 6 decide what a dimension remembers when an attribute changes, why bitemporal modelling separates when something was true from when you learned it, and how an overwrite quietly leaks the future into a training set.
- 08 Table Formats and the Lakehouse What a directory of Parquet files cannot do, how a metadata layer adds atomic commits and time travel on top of immutable object storage, and where the abstraction leaks.