Data Architecture
General material on structuring, storing and governing data.
10 to work through
-
intermediate
A dashboard query that took 200ms now takes 40 seconds. The table has grown to 200 million rows. Walk me through diagnosis and fix, including what you would not do.
3 min answer -
intermediate
A product catalogue page does 40,000 reads per second against a database that can serve 5,000. Walk me through the caching design, including what happens at 3 AM when the cache is empty.
2 min answer -
intermediate Multiple choice
A service writes to its database and then publishes an event to Kafka. Sometimes consumers see an event for a record that does not exist, and sometimes a record exists with no event. Why, and how do you fix it?
2 min answer -
intermediate
A visual discovery platform serves enormous read volume over user-curated collections, with writes that are rare by comparison. A team proposes sharding the primary database immediately. What lower-complexity options should be evaluated first, and what evidence would justify sharding?
2 min answer -
advanced
A B2B commerce platform aggregates inventory from many suppliers whose feeds arrive asynchronously with different frequencies and reliability. How should stale inventory, safety stock, reconciliation and supplier failures be handled?
2 min answer -
advanced
A feed-ranking system needs embeddings, counters and aggregates pushed from offline jobs to an online store at very high write throughput. How should such a derived-data serving system handle full rebuilds, incremental updates, versioning and reads during a swap?
2 min answer -
advanced
A multi-tenant SaaS product has outgrown one database. You must shard. How do you choose the partition key, and what makes this decision so expensive to get wrong?
2 min answer -
advanced
Discord stores trillions of messages. What is their partition key, and what problem does the second half of it solve?
2 min answer -
advanced
Netflix personalises an entire home page in the time a TV takes to draw a screen. How, given that scoring every title for every member on request is impossible?
2 min answer -
advanced Multiple choice
Uber indexes the world with hexagons rather than squares for surge pricing and dispatch. Why does the shape matter, and what is the general lesson?
2 min answer
25 terms in this topic
Bloom Filter
A compact probabilistic structure that answers "is this key definitely absent, or possibly present?" — no false negatives, tunable false positives.
conceptCache Invalidation
The problem of removing or refreshing cached data when the underlying source changes, and the reason caching is harder than it looks.
patternChange Data Capture
Publishing a stream of a database's row-level changes by reading its replication log, without modifying the application that owns it.
practiceClaim Provenance
Storing every externally sourced value with its timestamp, its source and a confidence derived from that source's history - so that downstream decisi…
patternCQRS
Separating the model used to change state from the model used to read it, so each can be optimised independently.
conceptData Lakehouse
A pattern that puts warehouse-style transactions, schema and governance on top of cheap open-format object storage.
conceptData Mesh
An organisational approach that gives domain teams ownership of their analytical data as a product, with a self-serve platform and federated governance.
practiceData Retention Policy
A defined rule for how long each class of data is kept, where, and what happens at the end of it.
conceptDatabase Index
A secondary structure that lets the engine find rows without scanning, trading write cost and storage for read speed.
practiceDenormalisation
Deliberately duplicating data across records to make reads cheap, accepting the write-time cost of keeping copies in step.
case-studyDiscord's Message Store Migrations
Discord moved from MongoDB to Cassandra to ScyllaDB as message volume grew from millions to trillions, each time for a specific and different reason.
conceptETL vs ELT
Whether data is transformed before loading into the target or after it, which decides where the compute happens and how much raw history you keep.
patternEvent Sourcing
Storing the full sequence of state-changing events as the system of record, and deriving current state by replaying them.
case-studyFigma's Postgres Sharding
Figma delayed sharding for years using replicas and vertical partitioning, then sharded Postgres horizontally without downtime using logical shards a…
practiceIndexing Strategy
Choosing the set of indexes a table carries by working backwards from its actual queries, and accepting the write cost that each one adds.
patternMaterialized View
A precomputed, stored result of a query, refreshed on a schedule or from a change stream, read instead of recomputing.
case-studyNetflix's Recommendation Architecture
Netflix splits personalisation into offline, nearline and online layers so that expensive computation happens ahead of time and the request path stay…
case-studyPinterest's MySQL Sharding
Pinterest sharded MySQL by embedding the shard ID inside every primary key, making any object's location computable from its ID alone with no lookup …
conceptQuery Plan
The database's chosen strategy for executing a query, and the first thing to look at when one is slow.
patternRead Replica
A copy of a database that receives changes from the primary and serves read-only queries, spreading read load.
case-studySalesforce's Metadata-Driven Multi-Tenancy
Salesforce serves every customer from shared infrastructure with a single physical schema, storing customer-specific data structures as metadata rath…
patternSharding
Splitting one dataset across multiple independent databases by a partition key, so that each holds a disjoint subset.
case-studyUber's H3 Spatial Index
Uber indexes the world with hexagons rather than squares, because uniform neighbour distance makes supply, demand and pricing computations correct as…
patternVersioned Dataset Swap
Publishing a regenerated dataset as a complete new version alongside the live one and flipping a serving pointer atomically, so readers never observe…
conceptWrite-Ahead Log
Recording every change to a durable sequential log before applying it, so that a crash can be recovered by replaying the log.
1 artifact you would hand over
Neighbouring topics
Relational Modelling
Normalisation, keys, constraints and the invariants a schema enforces.
NoSQL Stores
Key-value, document, wide-column and graph — what each buys and forbids.
Indexing
Designing indexes per query shape, and paying for them on every write.
Query Optimisation
Reading a plan, fixing statistics, and finding the real bottleneck.
Transactions & Isolation
ACID, isolation levels, and the anomalies each level permits.
Replication
Primaries, replicas, lag, and synchronous versus asynchronous durability.
Partitioning & Sharding
Splitting data across machines, and the one-way door of a partition key.
Caching Strategies
Cache-aside, read-through, write-through and where each belongs.
Cache Invalidation
Stampedes, penetration, staleness windows and versioned keys.
CQRS
Separating the write model from the read models that serve queries.
Event Sourcing
Storing the change log as the system of record, and what that costs forever.
Change Data Capture
Turning a database's replication log into a stream, and its coupling risk.
Data Warehousing
Dimensional modelling, star schemas and analytical workloads.
Data Lakes & Lakehouses
Open formats on object storage with transactional metadata on top.
ETL & ELT
Where transformation happens, and how much raw history you keep.
Streaming Data
Windowing, watermarks, late arrivals and exactly-once semantics.
Data Governance
Ownership, lineage, quality, catalogues and who may see what.
Data Lifecycle & Retention
How long data is kept, where it ages to, and how it is actually deleted.
Polyglot Persistence
Choosing a store per workload, and the operational cost of variety.