Event Streaming

Definition

An event stream is an append-only, partitioned, durable log. Consumers track their own position (offset), so the same event can be read by many consumers independently and re-read later. This is the property that distinguishes it from a queue, where a message is typically consumed once and removed.

Why it matters

The problem streaming solves is not throughput — it is integration topology. LinkedIn's adoption of a central log came from a combinatorial pain: every new system that needed profile or activity data added another bespoke point-to-point integration to every source system. Integrations grew roughly with the product of systems, not their sum.

Publishing to a log inverts the dependency. A producer writes once and does not know who consumes. A new consumer is added without touching any producer. That is the architectural purchase.

What is paid for it: eventual consistency everywhere downstream, schema governance as a permanent obligation (a published event is an API and breaking it breaks unknown consumers), consumer lag as a first-class operational metric, and a genuinely complex piece of infrastructure to operate.

Implementation patterns

• Partition by a key that carries the ordering requirement. Ordering is guaranteed only within a partition. Keying by user_id or order_id gives per-entity ordering, which is almost always what is actually needed. Global ordering requires one partition, which throws away parallelism.
• Consumer groups for horizontal scaling: each partition is assigned to one consumer in the group, so parallelism is capped by the partition count. Choose partition count for the parallelism you will need later, because increasing it changes key-to-partition mapping.
• Schema registry with compatibility rules. Additive changes only, enforced in CI.
• Events as facts, not commands. OrderPlaced, not SendConfirmationEmail. Facts have many legitimate consumers; commands have one, and belong in a queue.
• Retention long enough to replay. Replay is one of the main reasons to choose a log at all — it lets a new consumer build its state from history, and lets a broken consumer be fixed and re-run.

Failure scenarios

• A hot partition. One key — a celebrity user, a huge merchant — receives a disproportionate share of events, so one partition and therefore one consumer saturates while others idle.
• Consumer lag unmonitored until a downstream is hours stale and nobody noticed.
• A breaking schema change deployed by a producer team that could not enumerate its consumers.
• Rebalancing storms. A consumer that is slow to process is evicted from the group, triggering a rebalance, which pauses everyone, which makes more consumers slow.
• Streaming used for a request/response interaction, where the producer actually needs an answer. That produces correlation IDs, reply topics and timeouts — a synchronous call with extra steps and worse debugging.

Trade-offs

Choose a log when you have multiple independent consumers of the same facts, need replay, or need throughput and retention a database table cannot provide. Choose a simple queue or a database table when there is one consumer and one purpose. The operational cost of a streaming platform — brokers, partitions, quotas, upgrades, the on-call knowledge — is a standing tax that a small number of workflows will never repay.

The most common architectural error of the last decade is adopting a streaming platform for every workflow in an organisation that had two workflows needing it.

Interview question

"You need per-customer ordering and 100,000 events per second. How many partitions, keyed on what, and what happens when one customer generates 30% of the traffic?"