LinkedIn and the Origin of Kafka
Kafka was built to replace point-to-point data integration between many systems with a single durable log that any system could publish to and any number could read.
The problem
By around 2010 LinkedIn had many systems producing data — the site, databases, metrics, logs — and many consuming it: search, recommendations, the data warehouse, monitoring, newsfeed. Each producer-consumer pair was a bespoke pipeline.
This is the M×N integration problem. Adding a consumer meant touching every producer; every pipeline had its own delivery semantics, its own failure modes and its own operational burden; and nobody had a coherent view of the whole.
Existing messaging systems did not fit. Traditional brokers were built for low-volume, low-latency messaging with per-message acknowledgement and deletion after consumption — not for very high throughput, retention, and re-reading history.
The design
Kafka's insight was to make the log itself the abstraction, rather than the queue.
- Append-only, ordered, persistent log per partition — sequential disk writes, which are dramatically faster than the random writes brokers had assumed.
- Consumers track their own offsets. The broker does not track per-consumer state, which is what makes many independent consumers cheap and makes re-reading trivial.
- Retention by time or size, not by consumption. History exists, so a new consumer can rebuild its entire state from the beginning.
- Partitioning for horizontal scale, with ordering guaranteed within a partition.
M×N becomes M+N: everyone publishes to the log, everyone reads from it.
The architectural lessons
1. A shared, durable, replayable log is a different primitive from a queue, and the difference — retention and independent offsets — is what enables event sourcing, CQRS read models, CDC pipelines and stream processing.
2. Reframing the integration problem beat optimising the pipelines. The pipelines were not the problem; having N² of them was. This is the recurring shape of good architecture: change what the components are, not how fast they run.
3. Working with the hardware, not against it. Sequential I/O, page cache, zero-copy transfer. Kafka is fast largely because it stopped fighting the disk.
4. Log-centric thinking generalises. Jay Kreps' essay "The Log" argues that the same structure underlies database replication, distributed consensus and stream processing — a genuinely load-bearing idea rather than a product pitch.