LinkedIn built a change capture system so that derived stores — search, graph, caches — could stay current without every application dual-writing to them.

pattern

Log-Based CDC

Capturing changes by reading the database's own write-ahead log, which sees every change with no load on the source and no application involvement.

pattern

Query-Based CDC

Detecting changes by repeatedly querying for rows modified since the last run — simple, universally available, and lossy in specific ways.

Data Architecture

Neighbouring topics

Data Architecture

General material on structuring, storing and governing data.

10 quiz 31 cards 25 terms

Relational Modelling

Normalisation, keys, constraints and the invariants a schema enforces.

3 quiz 10 cards 4 terms

NoSQL Stores

Key-value, document, wide-column and graph — what each buys and forbids.

5 quiz 13 cards 4 terms

Indexing

Designing indexes per query shape, and paying for them on every write.

5 quiz 12 cards 6 terms

Query Optimisation

Reading a plan, fixing statistics, and finding the real bottleneck.

3 quiz 11 cards 4 terms

Transactions & Isolation

ACID, isolation levels, and the anomalies each level permits.

3 quiz 10 cards 4 terms

Replication

Primaries, replicas, lag, and synchronous versus asynchronous durability.

4 quiz 14 cards 6 terms

Partitioning & Sharding

Splitting data across machines, and the one-way door of a partition key.

6 quiz 12 cards 10 terms

Caching Strategies

Cache-aside, read-through, write-through and where each belongs.

7 quiz 15 cards 6 terms

Cache Invalidation

Stampedes, penetration, staleness windows and versioned keys.

3 quiz 7 cards 2 terms

CQRS

Separating the write model from the read models that serve queries.

4 quiz 10 cards 5 terms

Event Sourcing

Storing the change log as the system of record, and what that costs forever.

5 quiz 13 cards 4 terms

Data Warehousing

Dimensional modelling, star schemas and analytical workloads.

4 quiz 13 cards 3 terms

Data Lakes & Lakehouses

Open formats on object storage with transactional metadata on top.

4 quiz 11 cards 4 terms

ETL & ELT

Where transformation happens, and how much raw history you keep.

3 quiz 9 cards 3 terms

Streaming Data

Windowing, watermarks, late arrivals and exactly-once semantics.

3 quiz 10 cards 2 terms

Data Governance

Ownership, lineage, quality, catalogues and who may see what.

3 quiz 9 cards 4 terms

Data Lifecycle & Retention

How long data is kept, where it ages to, and how it is actually deleted.

4 quiz 11 cards 5 terms

Polyglot Persistence

Choosing a store per workload, and the operational cost of variety.

6 quiz 12 cards 4 terms

Change Data Capture

5 to work through

A logistics platform needs its operational database changes to reach a search index and an analytics warehouse. An engineer proposes writing to all three from the application. What is wrong, and what is the alternative?

A marketplace's search index lags several minutes behind catalogue updates and sellers complain their listings appear stale. Compare synchronous indexing, CDC pipelines and read-after-write patching at query time.

A downstream team needs to react to order changes. The order service can publish events, or they can consume CDC from its database. Which, and why?

A mobility platform wants operational database changes available to analytics and search within seconds, without adding load to the transactional databases. Design the change-data-capture pipeline and identify its principal failure modes.

A team plans to publish database row changes via CDC as the organisation's domain events. What is wrong with that and what would you propose?

5 terms in this topic

CDC Initial Snapshot