Streaming Data Architecture

Definition

A streaming architecture processes data as it arrives, maintaining state across an unbounded input rather than operating on a fixed dataset. The defining difficulty is not volume; it is that the data is never complete.

The two clocks

• Event time — when the thing happened.
• Processing time — when the system saw it.

They differ, sometimes by seconds and sometimes by hours: a mobile device was offline, a partition was slow, a consumer was restarted. Every windowed aggregation must therefore answer: how long do we wait for late data before emitting a result, and what do we do with data that arrives after that?

The available answers — a watermark that estimates completeness, a grace period, and a policy for late arrivals (drop, side-output, or emit a correction) — are business decisions, not technical ones. A dashboard can drop; a driver payout cannot.

Industry example

Real-time delivery marketplaces make this concrete. A DoorDash-style dispatch system must decide, continuously, which courier should take which order, using state that is changing every few seconds: courier locations, order readiness, traffic, restaurant load.

Three characteristics follow:

Freshness has a hard business value. A dispatch decision made on 60-second-old location data is measurably worse than one made on 5-second-old data, in a way that shows up in delivery time and cancellation rate. That justifies streaming where most "real-time" requirements do not.

Some state must be strongly consistent while most is approximate. An assignment is exclusive — two couriers must not be given the same order — so that specific decision needs a real consistency mechanism. Everything feeding it, including location and demand estimates, is inherently approximate and must be treated that way.

The system must degrade rather than stop. If the streaming layer is impaired, dispatch must fall back to a simpler heuristic on staler data, not stop assigning orders. A marketplace that stops matching is a marketplace that is down.

Implementation patterns

• Windowing chosen deliberately — tumbling, sliding or session — matched to the question being asked.
• Watermarks with an explicit lateness policy, documented, not defaulted.
• Checkpointed state so a failed operator resumes rather than recomputes.
• Idempotent sinks, because at-least-once processing will re-emit.
• A batch path for correctness alongside the streaming path where the numbers must eventually be exact — reconciliation rather than a full second pipeline, wherever possible, since maintaining two implementations of the same logic guarantees they diverge.

Failure scenarios

• Unbounded state. A keyed aggregation with no expiry grows until the job dies. State TTL is mandatory, not optional.
• Watermark stalled by one idle partition, so results stop emitting entirely and no error is raised.
• Late data silently dropped, so totals are quietly wrong and no metric shows it.
• Streaming chosen where hourly batch would have been indistinguishable to users and an order of magnitude cheaper and simpler.

Interview question

"Your streaming aggregation stops emitting results but the job is healthy and consuming. What is the most likely cause?"