Data Architecture

tool

Bloom Filter

A compact probabilistic structure that answers "is this key definitely absent, or possibly present?" — no false negatives, tunable false positives.

concept

Cache Invalidation

The problem of removing or refreshing cached data when the underlying source changes, and the reason caching is harder than it looks.

pattern

Change Data Capture

Publishing a stream of a database's row-level changes by reading its replication log, without modifying the application that owns it.

pattern

CQRS

Separating the model used to change state from the model used to read it, so each can be optimised independently.

concept

Data Lakehouse

A pattern that puts warehouse-style transactions, schema and governance on top of cheap open-format object storage.

concept

Data Mesh

An organisational approach that gives domain teams ownership of their analytical data as a product, with a self-serve platform and federated governance.

practice

Data Retention Policy

A defined rule for how long each class of data is kept, where, and what happens at the end of it.

concept

Database Index

A secondary structure that lets the engine find rows without scanning, trading write cost and storage for read speed.

practice

Denormalisation

Deliberately duplicating data across records to make reads cheap, accepting the write-time cost of keeping copies in step.

case-study

Discord's Message Store Migrations

Discord moved from MongoDB to Cassandra to ScyllaDB as message volume grew from millions to trillions, each time for a specific and different reason.

concept

ETL vs ELT

Whether data is transformed before loading into the target or after it, which decides where the compute happens and how much raw history you keep.

pattern

Event Sourcing

Storing the full sequence of state-changing events as the system of record, and deriving current state by replaying them.

case-study

Figma's Postgres Sharding

Figma delayed sharding for years using replicas and vertical partitioning, then sharded Postgres horizontally without downtime using logical shards a…

practice

Indexing Strategy

Choosing the set of indexes a table carries by working backwards from its actual queries, and accepting the write cost that each one adds.

pattern

Materialized View

A precomputed, stored result of a query, refreshed on a schedule or from a change stream, read instead of recomputing.

case-study

Netflix's Recommendation Architecture

Netflix splits personalisation into offline, nearline and online layers so that expensive computation happens ahead of time and the request path stay…

case-study

Pinterest's MySQL Sharding

Pinterest sharded MySQL by embedding the shard ID inside every primary key, making any object's location computable from its ID alone with no lookup …

concept

Query Plan

The database's chosen strategy for executing a query, and the first thing to look at when one is slow.

pattern

Read Replica

A copy of a database that receives changes from the primary and serves read-only queries, spreading read load.

case-study

Salesforce's Metadata-Driven Multi-Tenancy

Salesforce serves every customer from shared infrastructure with a single physical schema, storing customer-specific data structures as metadata rath…

pattern

Sharding

Splitting one dataset across multiple independent databases by a partition key, so that each holds a disjoint subset.

case-study

Uber's H3 Spatial Index

Uber indexes the world with hexagons rather than squares, because uniform neighbour distance makes supply, demand and pricing computations correct as…

concept

Write-Ahead Log

Recording every change to a durable sequential log before applying it, so that a crash can be recovered by replaying the log.