LinkedIn Databus: Change Capture as a Product
also called Databus
LinkedIn built a change capture system so that derived stores — search, graph, caches — could stay current without every application dual-writing to them.
The problem
A profile update at LinkedIn must reach many derived systems: the search index, the social graph, the recommendation features, caches, the data warehouse.
The naive approach has the application write to each. That is the dual-write problem at scale, and it fails in a specific way: the application updates the database and then writes to the search index, the process dies in between, and the two are now permanently inconsistent with nothing detecting it. Multiply by five derived stores and the inconsistency is continuous.
It also couples every application to every derived system: adding a new consumer means changing every producer.
What they did
Databus captures changes from the source database's commit log and publishes them as an ordered, replayable stream that any number of consumers subscribe to independently.
Three properties matter. Changes are captured from the committed log, so an event exists if and only if the transaction committed — which eliminates the dual-write gap entirely. Consumers track their own position, so a slow or failed consumer does not affect the source or other consumers. And the stream is replayable from a point in the past, so a consumer can rebuild its state from scratch after a defect.
The trade-off
The most consequential operational hazard is that a stalled consumer causes the source database to retain log segments, and the failure mode is the source system's disk filling — a data pipeline problem becoming a production database outage. This needs its own alerting and it is routinely missing.
There is also a coupling to the source schema: consumers see the database's internal model, so a refactor upstream breaks them. This is what data contracts and an explicit published event schema address, and it is the reason mature implementations transform the raw change stream into a stable domain event before publishing widely.
The transferable lesson
If you find yourself writing the same change to two systems in application code, you have a correctness problem, not an integration one. The fixes are change data capture from the log, or the transactional outbox where the event is written in the same transaction as the data.
Both give a single source of truth for what happened, and both make adding a consumer a subscription rather than a change to every producer.