LinkedIn: Kafka and the Unified Log

The problem

By around 2010 LinkedIn had many systems producing data — the profile database, activity tracking, search indexes, the recommendation engines, the data warehouse, monitoring — and many systems consuming it. Each producer-consumer pair had been connected with a bespoke pipeline.

The mathematics of that arrangement are unforgiving. With n systems needing to exchange data, you tend towards n² integrations, each with its own format, its own failure modes, its own backfill story and its own owner. Adding a system means building integrations to everything it needs. Changing a producer means finding every consumer, and nobody has the list.

What they did

Kafka inverted the topology. Producers write once to a durable, ordered, replayable log; consumers read at their own pace and track their own position. The integration count drops from n² to n, and the log becomes the organisation's shared substrate for data in motion.

Three properties did the real work. Durability with retention, so a consumer can be added later and read history it was not present for. Independent consumer offsets, so a slow or failed consumer does not affect producers or other consumers. And ordering within a partition, which is what makes the log usable for state replication rather than only for notifications.

The trade-off

The log becomes a shared dependency of very high consequence, with its own operational discipline — partition and key design, retention sizing, consumer lag monitoring, schema compatibility. Teams that adopt it as "a queue" and skip the schema registry and key design discover the costs later, usually as an ordering bug or a hot partition.

And it moves the coupling rather than removing it: consumers still depend on the meaning of the events, which no infrastructure fixes.

The transferable lesson

Count your integrations. If adding a system requires building connections to several others, and changing a producer requires finding unknown consumers, the topology is the problem and no amount of improving individual pipelines addresses it.

The log pattern applies well below LinkedIn's scale. A dozen systems exchanging data point-to-point is already past the point where a shared event backbone is cheaper — and the retention property, which lets a new consumer bootstrap from history, is usually the benefit that convinces people.