A logistics platform feeds its warehouse from operational databases by change data capture. What must the pipeline design handle?

What it must handle

• Ordering per key. Changes to the same row must be applied in order, which comes free from a partitioned log if the partition key is the primary key — and is broken if the consumer processes a partition concurrently.
• Deletes. A hard delete in the source must be represented, and a warehouse that ignores deletes silently accumulates records that no longer exist — producing counts that drift upward and reports that are wrong in a direction nobody checks.
• Schema changes. A column added, renamed, retyped or dropped in the source. CDC exposes the operational schema as a public interface, so a rename breaks consumers who never declared a dependency — which is CDC's real cost and it is organisational rather than technical.
• Snapshot plus stream convergence. Initial load and ongoing changes must join without a gap: start the stream first from a known position, then snapshot, then apply the stream from that position with idempotent handling. If the snapshot runs first, changes during it are lost silently.
• Idempotency, since the stream is at-least-once and will replay after a failover.

The consumer-side requirement

Merge semantics in the warehouse. A change stream produces inserts, updates and deletes, and the destination must apply them — which for a columnar store is an expensive operation and typically means periodic merge rather than continuous application.

That merge cadence is the freshness dial, and it is a cost-versus-freshness decision that should be made per table rather than globally.

The operational requirement

• Lag monitoring per table, with an owner. A stalled stream produces stale data with no error.
• A replay path, so a consumer bug can be corrected by reprocessing from a known offset rather than by a bespoke backfill.
• Isolation per source, so one operational database's problem does not stall the whole platform.

The alternative worth considering

A transactional outbox for the events that are a designed contract, alongside CDC for the bulk replication. The outbox publishes what you chose to emit with a schema you own; CDC publishes every row change.

The outbox gives a stable contract and CDC gives completeness, and running both for different purposes is common and correct.