CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
5 to work through
-
intermediate
A CDC connector on a 4 TB Postgres table is restarted with a changed configuration during business hours, and the operator does not realise this triggers a new initial snapshot. What happens over the next three hours, upstream and downstream?
3 min answer -
advanced
A catalogue search index is rebuilt every night by a full extract of a 60-million-row product table. You must move it onto a change-data-capture stream with no search downtime and no window where the index is silently wrong. Sequence it so each step is reversible, and say where the point of no return is.
3 min answer -
advanced
A platform publishes database changes as a stream for many consumers. What must the published stream provide that the raw change log does not?
2 min answer -
advanced
A service rebuilds its cache from a compacted topic after being offline for two weeks. It now holds records that were deleted. Why?
2 min answer -
advanced
A team proposes exposing their service's database change stream via CDC so other teams can consume it, avoiding the work of building an event API. What is your assessment?
2 min answer
3 terms in this topic
Log Compaction
Retaining only the most recent value for each key in a topic, so the log becomes a durable snapshot of current state rather than a bounded window of …
conceptReplication Slot Retention
The write-ahead log a database must keep because a change-data-capture consumer has not yet confirmed it, which turns a slow or stopped CDC pipeline …
conceptTransaction Log Stream
Change events derived from a database's write-ahead log — faithful to the table's mutations and to its internal model rather than to the business's.
Neighbouring topics
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.