intermediate 2 min answer

A logistics platform needs its operational database changes to reach a search index and an analytics warehouse. An engineer proposes writing to all three from the application. What is wrong, and what is the alternative?

portercdcdual-writeoutboxconsistency
Show the full answer Hide the answer

What is wrong with the dual write

The application writes to the database, then to the search index, then to the warehouse. There is no transaction spanning them, so any failure between the writes leaves the systems disagreeing — and the disagreement is permanent, because nothing detects or repairs it.

It is also unordered under concurrency: two updates to the same record can reach the database in one order and the search index in another, so the index ends up holding the older value with no error anywhere.

And it couples availability: if the search index is down, either the write fails (durability now depends on a search cluster) or the index is silently skipped (permanent divergence).

The alternative

Change data capture. The database's replication log is read by a connector and published as an ordered stream of changes, which downstream systems consume independently.

  • One source of truth, and every derived system is explicitly derived.
  • Ordering comes free from the log, per key.
  • Consumers are decoupled: the search index being down delays the index and does not affect writes.
  • Replayable, so rebuilding an index means resetting an offset rather than writing a bespoke backfill.
  • No application change to add a consumer, which is the property that makes it scale organisationally.

The intermediate option

If CDC infrastructure is not available, the transactional outbox achieves the essential property with much less machinery: the application writes the state change and an outbox row in one local transaction, and a relay publishes outbox rows afterwards. It gives atomicity between the change and its publication, which is the actual requirement.

The difference is that the outbox captures events you chose to emit, while CDC captures every change. The outbox is often better for that reason — it gives you a designed contract rather than exposing your schema as a public interface, and downstream consumers of raw CDC become coupled to your table structure.

What CDC still does not solve

Consumers must be idempotent, because the stream is at-least-once. And schema evolution becomes a cross-system concern: a column rename in the operational database now breaks downstream consumers who never knew they depended on it. That is the real cost of CDC and it is organisational rather than technical.