Concept library
1134 concepts across 20 domains and 101 tracks. Each track is a coherent sequence — read it top to bottom or dip in wherever the gap is.
All domains
01Foundations
02Transformer Internals
03Training & Fine-Tuning
04Reinforcement Learning
05Inference, Systems & Hardware
06Applied LLM Engineering
07Reasoning, Evaluation & Safety
08Multimodal & Applications
09Classical ML & Statistical Learning
10Causal Inference & Experimentation
11Time Series & Forecasting
12Graphs, Recommenders & Structured Data
13Generative Modelling Beyond Transformers
14Efficiency, Compression & Edge AI
15Search & Information Retrieval
16Data & Feature Engineering
17MLOps & Platform Engineering
18Security, Privacy & Adversarial ML
19Governance, Risk & Responsible AI
20Human-AI Interaction, Product & Economics
16
Data & Feature Engineering
The pipelines, formats and contracts that decide whether a model ever sees correct inputs.
5tracks
37concepts
381cards
4.5hreading
Data Modelling & Storage Columnar formats, table formats and the lakehouse, partitioning, and modelling choices that decide query cost. 11 concepts · 95 cards
- 01 Arrow and Zero-Copy Columnar Interchange The difference between a storage format and a memory format, why Arrow eliminated a conversion step that cost more than the query, and where copies still happen despite the name.
- 02 Columnar Formats and Why They Win What changes when values of the same column sit next to each other on disk, why that makes compression an order of magnitude better, and the workloads where row storage is still correct.
- 03 Compaction and the Small File Problem Why streaming ingestion destroys query performance on a table that is otherwise perfectly healthy, what compaction actually costs, and the write-amplification trap in naive compaction policies.
- 04 Copy-on-Write versus Merge-on-Read The two ways to change one row inside an immutable file, why the choice is really a decision about where to spend amplification, and what deletion vectors changed.
- 05 Dimensional Modelling and the Case for Wide Tables Why star schemas were designed for a storage economics that no longer holds, what they still provide, and how to decide between a normalised model and one wide denormalised table.
- 06 Object Storage Semantics for Analytics Why a lakehouse built on S3 behaves nothing like one built on a filesystem, and the four properties of object storage that decide table layout, commit protocols and query cost.
- 07 Slowly Changing Dimensions and History How SCD types 1, 2, 3 and 6 decide what a dimension remembers when an attribute changes, why bitemporal modelling separates when something was true from when you learned it, and how an overwrite quietly leaks the future into a training set.
- 08 Table Formats and the Lakehouse What a directory of Parquet files cannot do, how a metadata layer adds atomic commits and time travel on top of immutable object storage, and where the abstraction leaks.
- 09 Partitioning, Clustering and File Sizing The three physical layout decisions that determine query cost, why partitioning on a high-cardinality column is the classic disaster, and how to reason about the right file size.
- 10 Predicate Pushdown and Column Statistics How a query engine avoids reading data it can prove is irrelevant, the hierarchy of pruning from partition to page, and why statistics that exist can still be useless.
- 11 Storage Layout for ML Training Reads Why the access pattern of a training loop is the one pattern the lakehouse was not designed for, and how sharded sequential formats trade shuffle quality for two orders of magnitude fewer requests.
Batch & Streaming Pipelines Event time versus processing time, watermarks, exactly-once semantics, backfills and orchestration. 7 concepts · 74 cards
- 01 Backfills and Reprocessing Why recomputing history is a different operation from running a pipeline fast, the resource and correctness hazards specific to it, and the design choices that make a pipeline backfillable at all.
- 02 Change Data Capture and the Outbox Pattern Why writing to a database and a message broker from the same request cannot be made atomic, how log-based change data capture turns the database's own commit log into the event stream, and how the outbox pattern keeps that stream a contract rather than a leak of internal schema.
- 03 Event Time Versus Processing Time The two clocks every streaming system has, why using the wrong one produces results that change on replay, and the skew that makes correctness a latency tradeoff.
- 04 Orchestration, Dependencies and Idempotency What an orchestrator is actually for, why retries are the whole reason tasks must be idempotent, and the difference between scheduling on time and scheduling on data availability.
- 05 Stream Windowing: Tumbling, Sliding and Session How a windowing function assigns each event on an unbounded stream to finite groups, why the three standard shapes differ in state cost and in whether their boundaries are shared across keys, and how late data makes session windows merge after results have been emitted.
- 06 Exactly-Once Semantics, Precisely What "exactly once" actually guarantees, why it is a statement about effects rather than about deliveries, and the two mechanisms that provide it.
- 07 Watermarks and Allowed Lateness The heuristic that lets a streaming system decide a window is complete, why it is always wrong in one of two directions, and how triggers and lateness policies turn a single answer into a sequence of refinements.
Feature Stores Offline-online parity, point-in-time correctness, materialisation, and the failure they exist to prevent. 6 concepts · 71 cards
- 01 Feature Reuse, Discovery and Ownership The organisational argument for a feature store, why reuse is harder than it sounds, and the governance problems that appear once several teams depend on one definition.
- 02 Materialisation and the Online Store How features get from an analytical table into a millisecond-latency lookup, the freshness-cost tradeoff each materialisation strategy makes, and why the online store's data model is nothing like the offline one.
- 03 Embeddings as Managed Features Why an embedding stored in a feature store is only meaningful alongside the exact encoder that produced it, what an encoder upgrade costs in backfill and downstream retraining, and how backward-compatible training and learned alignment trade model quality for avoiding that cost.
- 04 Offline-Online Parity and Training-Serving Skew Why the same feature computed by two pipelines is rarely the same number, the categories of divergence, and the architectural choices that eliminate rather than manage the problem.
- 05 Point-in-Time Correctness The join that a feature store exists to get right, why a naive join on entity ID leaks the future into training labels, and what an as-of join costs to compute.
- 06 Streaming Aggregations for Real-Time Features How to compute a sliding-window count over millions of entities within a serving latency budget, why exact windows are usually unaffordable, and the approximations that are safe.
Data Quality & Contracts Expectations and assertions, schema evolution, producer-consumer contracts, and detecting silent corruption. 6 concepts · 69 cards
- 01 Data Contracts as Producer Obligations Why moving the schema definition to the producer changes the economics of data quality, what a contract must contain beyond field types, and the organisational conditions under which contracts actually work.
- 02 Expectations, Assertions and Where to Put Them The categories of check a data pipeline can make, why the placement of a check matters more than the check itself, and the failure mode of testing only what is easy to test.
- 03 Testing Data Transformations How to test the logic of a SQL or dataframe transformation rather than the data flowing through it, using fixture-based unit tests, property-based tests that state invariants, and reconciliation checks that compare what went in with what came out.
- 04 Anomaly Detection on Data Pipelines Why learned thresholds beat static ones for volume and freshness, the base-rate problem that makes naive alerting useless, and how to structure alerts so that people still read them after six months.
- 05 Detecting Silent Corruption The failures that produce structurally valid, plausible, wrong data, why per-row checks cannot find them, and the reconciliation techniques that can.
- 06 Schema Evolution and Compatibility Modes The precise definitions of backward, forward and full compatibility, why the direction depends on whether you deploy producers or consumers first, and the changes that are always breaking.
Data Governance & Lineage Catalogues, column-level lineage, retention and deletion, access control and provenance for training corpora. 7 concepts · 72 cards
- 01 Catalogues, Metadata and Discovery The three kinds of metadata a catalogue holds, why the technical layer is the only one that stays accurate for free, and what makes a catalogue get used rather than abandoned.
- 02 Column-Level Lineage What lineage answers that a dependency graph cannot, why column granularity changes the questions you can ask, and the parsing problem that makes it hard to capture correctly.
- 03 Data Classification and Sensitivity Labelling Why a sensitivity scheme is only as good as how labels propagate through joins, aggregates, embeddings and models, how the high-water-mark rule makes propagation mechanical, and the over-classification and inference failures that quietly turn labels into noise.
- 04 Access Control for Analytical Data Why role-based access breaks down on a data platform, how attribute-based policies and dynamic masking replace it, and the derived-table hole that undermines both.
- 05 Consent and Purpose Limitation for Training Data Why data collected for one purpose cannot simply be reused to train a model under GDPR, how the Article 6(4) compatibility test and the legitimate-interest route work in practice, what EDPB Opinion 28/2024 and the courts have said, and where the law stands as of September 2026.
- 06 Provenance for Training Corpora Why the governance question for training data is harder than for analytical data, what a defensible provenance record contains, and the specific obligations that make it necessary.
- 07 Retention, Deletion and the Right to Erasure Why deleting a row is not deleting data, the specific places copies survive in a modern data platform, and the architectures that make erasure tractable.