Log-Based Ingestion
Building the pipeline around a database's own change log — an initial snapshot followed by a continuous delta stream, with the two stitched together.
Reading the transaction log rather than querying the table gives every change in
commit order, including deletes, with no load on the source and no dependence on a
reliable updated_at column.
The engineering is in the seam. A pipeline needs a consistent snapshot of the table plus every change since a known log position, and the two must join without gap or duplication. The snapshot takes time; changes occur during it. The standard approach records the log position before snapshotting and replays from there, relying on the downstream apply being idempotent so re-applied changes are harmless.
Three details decide whether it works in production. Log retention on the source must exceed the worst plausible consumer outage, or recovery means a full re-snapshot. Schema changes appear in the stream and the pipeline must handle them rather than fail on them. And the ordering guarantee holds only within a partition, so the partition key must keep all changes to one row together — otherwise an update can overtake the insert that created it.