advanced 2 min answer

A platform publishes database changes as a stream for many consumers. What must the published stream provide that the raw change log does not?

cdccontractscouplingtransformationebaydesign
Show the full answer Hide the answer

What the raw log does not provide

A stable contract. The raw change log carries the operational database's internal schema — column names, types and structures the owning team considers private and changes freely.

Publishing it directly means every downstream consumer depends on an internal schema, so a refactor the owning team regards as internal breaks consumers they have never met. The database schema has become a public interface nobody agreed to publish.

What the published stream must add

1. A translated, versioned contract. Events shaped in domain terms rather than table terms, with a schema registry enforcing compatibility as a build gate. The owner can then refactor the underlying tables freely as long as the translation is maintained.

2. Semantic events where possible. "Order placed", "order cancelled" rather than "row in orders table changed, with these before and after images". Consumers care about what happened in the business, and deriving that from column deltas puts the same logic in every consumer.

3. Transaction grouping where cross-table consistency matters. Raw CDC emits per-row changes, so a consumer can observe a state that never existed atomically. If that matters, the published stream must group by transaction and emit a coherent event.

4. Deletes represented explicitly, including whether a delete is a real removal or a soft delete the operational model treats differently.

5. Sensitive fields filtered or tokenised at publication, so classification is enforced at the boundary rather than by every consumer.

6. Documented ordering guarantees — per key, not global — so consumers do not assume more than is provided.

The cost of the translation layer

It is real: another component to operate, a place logic can be wrong, and additional latency. The alternative is a permanent coupling between an operational schema and an unbounded set of consumers, which is what prevents the owning team from ever changing anything.

The governance point

The published stream is a data product with an owner, not a side effect of a database. It needs a contract, a consumer registry, a deprecation policy and a freshness SLI — the same obligations as any published interface.