Term Kind Topic What it is
Backfill practice ETL & ELT Re-running a pipeline over historical periods to populate new data or correct a past error, and the operation that proves whether a pipeline is well designed.
Bloom Filter tool Data Architecture A compact probabilistic structure that answers "is this key definitely absent, or possibly present?" — no false negatives, tunable false positives.
Bloom Filter Cache Guard pattern Caching Strategies Placing a Bloom filter in front of an expensive lookup so that keys which certainly do not exist never reach it.
Cache Invalidation concept Data Architecture The problem of removing or refreshing cached data when the underlying source changes, and the reason caching is harder than it looks.
Cache Penetration concept Cache Invalidation Repeated lookups for keys that do not exist, which miss the cache every time by definition and pass straight through to the store.
Cache Stampede Dog-piling, Thundering Herd on Cache concept Cache Invalidation Many concurrent requests missing on the same expired key and all recomputing it simultaneously, converting one expiry into a load spike.
Cardinality Estimation concept Query Optimisation The planner's prediction of how many rows each step of a query will produce — the input that determines every other choice it makes.
CDC Initial Snapshot concept Change Data Capture The consistent full copy taken when a CDC pipeline starts, before streaming begins — and the step that determines whether the target is correct.
Change Data Capture CDC pattern Data Architecture Publishing a stream of a database's row-level changes by reading its replication log, without modifying the application that owns it.
Covering Index Index-Only Scan concept Indexing An index that contains every column a query needs, so the query is answered from the index without reading the table at all.
CQRS Command Query Responsibility Segregation pattern Data Architecture Separating the model used to change state from the model used to read it, so each can be optimised independently.
Crypto-Shredding Cryptographic Erasure pattern Data Lifecycle & Retention Encrypting each subject's data with its own key and destroying that key to render the data permanently unreadable, achieving deletion without deleting.
Data Catalog tool Data Governance A searchable inventory of datasets with their schema, owner, meaning, freshness, quality and classification.
Data Contract practice Data Governance An explicit, versioned, enforced agreement between a data producer and its consumers about schema, semantics, quality and change policy.
Data Lakehouse concept Data Architecture A pattern that puts warehouse-style transactions, schema and governance on top of cheap open-format object storage.
Data Lineage practice Data Governance A record of where each dataset came from, what transformed it, and what depends on it — traced at table and ideally column level.
Data Mesh concept Data Architecture An organisational approach that gives domain teams ownership of their analytical data as a product, with a self-serve platform and federated governance.
Data Retention Policy practice Data Architecture A defined rule for how long each class of data is kept, where, and what happens at the end of it.
Database Index concept Data Architecture A secondary structure that lets the engine find rows without scanning, trading write cost and storage for read speed.
Database per Service pattern Polyglot Persistence Each service owning its own datastore, with no other service reading or writing it directly.
Denormalisation practice Data Architecture Deliberately duplicating data across records to make reads cheap, accepting the write-time cost of keeping copies in step.
Discord's Message Store Migrations case-study Data Architecture Discord moved from MongoDB to Cassandra to ScyllaDB as message volume grew from millions to trillions, each time for a specific and different reason.
Document Store tool NoSQL Stores A store that keeps semi-structured documents — typically JSON — retrievable by key and queryable by their contents.
ETL vs ELT concept Data Architecture Whether data is transformed before loading into the target or after it, which decides where the compute happens and how much raw history you keep.
Event Sourcing pattern Data Architecture Storing the full sequence of state-changing events as the system of record, and deriving current state by replaying them.
Event Store tool Event Sourcing An append-only store of domain events organised into per-entity streams, serving as the system of record rather than as a log beside it.
Event Upcasting pattern Event Sourcing Transforming an old event's stored form into the current shape as it is read, so historical events remain replayable after the schema changes.
Figma's Postgres Sharding case-study Data Architecture Figma delayed sharding for years using replicas and vertical partitioning, then sharded Postgres horizontally without downtime using logical shards and a proxy layer.
Foreign Key Constraint concept Relational Modelling A database-enforced rule that a referencing value must exist in the referenced table — referential integrity that no application bug can violate.
Graph Database tool NoSQL Stores A store whose first-class citizens are nodes and the relationships between them, making multi-hop traversal cheap.
Hot Partition Hot Shard, Hot Key concept Partitioning & Sharding One partition receiving disproportionate traffic, so the system saturates at a fraction of its aggregate capacity.
Hot, Warm and Cold Data concept Data Lifecycle & Retention Classifying data by how frequently and how urgently it is accessed, so each tier can be stored on media priced for that access pattern.
Idempotent Pipeline practice ETL & ELT A pipeline whose task can be re-run for the same input window any number of times and produce the same result.
Index Selectivity metric Indexing The fraction of rows a predicate eliminates — the property that determines whether an index is worth using at all.
Indexing Strategy practice Data Architecture Choosing the set of indexes a table carries by working backwards from its actual queries, and accepting the write cost that each one adds.
Join Strategies concept Query Optimisation The three ways a database combines two row sets — nested loop, hash join and merge join — and the conditions under which each is correct.
Log-Based CDC pattern Change Data Capture Capturing changes by reading the database's own write-ahead log, which sees every change with no load on the source and no application involvement.
Materialized View pattern Data Architecture A precomputed, stored result of a query, refreshed on a schedule or from a change stream, read instead of recomputing.
Multi-Leader Replication Multi-Master, Active-Active Replication pattern Replication Accepting writes at more than one node and replicating between them, which removes the single-primary bottleneck and introduces write conflicts.
Netflix's Recommendation Architecture case-study Data Architecture Netflix splits personalisation into offline, nearline and online layers so that expensive computation happens ahead of time and the request path stays fast.
Normalisation Normal Forms practice Relational Modelling Organising a schema so each fact is stored exactly once, removing the update anomalies that duplication creates.
OLTP vs OLAP concept Data Warehousing Two workload shapes with opposite requirements — many small indexed transactions versus few large scans and aggregations — which is why they belong in different stores.
Open Table Format Apache Iceberg, Delta Lake, Apache Hudi protocol Data Lakes & Lakehouses A metadata layer over files in object storage that supplies ACID transactions, schema evolution and time travel — the thing that turns a data lake into a lakehouse.
Operational vs Analytical Store concept Polyglot Persistence The separation between the store serving the application's transactions and the one serving reporting and analysis, and the mechanism connecting them.
Partial Index Filtered Index concept Indexing An index built over only the rows matching a predicate, so it is far smaller and cheaper to maintain than a full index.
Pinterest's MySQL Sharding case-study Data Architecture Pinterest sharded MySQL by embedding the shard ID inside every primary key, making any object's location computable from its ID alone with no lookup service.
Pipeline Orchestration practice ETL & ELT Coordinating the execution of data tasks by dependency rather than by clock, with retries, backfill and observability built in.
Projection pattern CQRS The process that consumes changes from the write side and maintains a read model, and the component where most CQRS bugs live.
Query Plan Execution Plan, EXPLAIN concept Data Architecture The database's chosen strategy for executing a query, and the first thing to look at when one is slow.
Query-Based CDC Polling CDC, Timestamp-Based CDC pattern Change Data Capture Detecting changes by repeatedly querying for rows modified since the last run — simple, universally available, and lossy in specific ways.
Read Model Query Model, Projection Store pattern CQRS A data structure shaped for a specific query rather than for the domain, maintained separately from the write model.
Read Replica pattern Data Architecture A copy of a database that receives changes from the primary and serves read-only queries, spreading read load.
Replication Lag metric Replication How far behind a replica is, measured in time or in log position — the quantity that determines how stale a replica read can be.
Right to Erasure Right to be Forgotten concept Data Lifecycle & Retention A data subject's right to have their personal data deleted, and an obligation that reaches every copy an architecture has created.
Salesforce's Metadata-Driven Multi-Tenancy case-study Data Architecture Salesforce serves every customer from shared infrastructure with a single physical schema, storing customer-specific data structures as metadata rather than as separate tables.
Schema Evolution concept Data Lakes & Lakehouses Changing a table's structure over time while keeping existing data readable and existing consumers working.
Sharding Horizontal Partitioning pattern Data Architecture Splitting one dataset across multiple independent databases by a partition key, so that each holds a disjoint subset.
Slowly Changing Dimension SCD pattern Data Warehousing A strategy for handling attributes that change over time, deciding whether history is preserved and how facts attach to the correct version.
Snapshotting pattern Event Sourcing Periodically storing an aggregate's computed state so it can be loaded without replaying its entire event history.
Star Schema pattern Data Warehousing A dimensional model with one central fact table of measurements surrounded by denormalised dimension tables describing them.