Data Platform Architecture
The analytical estate: storage layout, ingestion, transformation and the compute that reads it.
20 topics in this area.
Each topic is the unit content is filed against. Topics with nothing in them yet are shown too — they are the honest edge of the curriculum rather than a gap being hidden.
Data Platform Architecture
General material on designing the analytical data estate end to end.
No content yetMedallion Architecture
Bronze, silver and gold layers, and what each layer is allowed to guarantee.
Open Table Formats
Iceberg, Delta and Hudi — transactions, snapshots and time travel over object storage.
No content yetWarehouse, Lake & Lakehouse
Three answers to where analytical data lives, and the workloads that separate them.
No content yetStorage Layout & Partitioning
Partition keys, clustering, and the scan the query planner is left able to skip.
No content yetFile Formats & Compaction
Columnar formats, the small-file problem, and the maintenance nobody schedules.
No content yetIngestion Patterns
Full load, incremental, append-only and merge, and the source system each one suits.
No content yetCDC Pipeline Design
Building on a change stream: snapshot plus delta, tombstones, and merge into the target.
No content yetBatch Orchestration
DAGs, dependencies, retries, and the difference between a schedule and an orchestration.
No content yetWorkflow Schedulers
Airflow, Dagster and their kin — where the control plane sits and what it can recover.
No content yetTransformation Frameworks
Declarative SQL transformation with tests, lineage and versioned models.
No content yetDimensional Modelling
Facts, dimensions, grain, and the star schema's continued relevance.
No content yetData Vault Modelling
Hubs, links and satellites, and the auditability and load parallelism they buy.
No content yetSlowly Changing Dimensions
Overwriting, versioning or timestamping attribute history, and the reporting each enables.
No content yetAnalytics Cost Control
Scanned bytes, idle warehouses, and the query nobody knew was running hourly.
No content yetWorkload Isolation
Keeping an analyst's query off the pipeline's compute, and both off the dashboard's.
No content yetData Platform Tenancy
Multiple domains on shared storage and compute, with separable access and cost.
No content yetReverse ETL
Pushing modelled analytical data back into operational systems, and who owns it then.
No content yetData Virtualisation
Querying across sources without moving data, and the performance ceiling that imposes.
No content yetWarehouse Migration
Moving off a legacy warehouse with thousands of reports pointed at it.
No content yet