Data View design intermediate

Streaming Topology Diagram

Topics, partitions, processors and state stores in one picture, with the retention and keying decisions that determine whether it can be replayed.

Data Platform Build AI / ML Platform Integration Programme Kafka Cloud-Agnostic Cross-Industry Engineering Operations & SRE

flowchart LR
  src1[("Orders DB")] --> cdc["CDC connector"]
  cdc --> t1[["orders.raw<br/><i>12 parts · key: orderId<br/>retain 7d</i>"]]
  src2["Clickstream"] --> t2[["events.clicks<br/><i>36 parts · key: sessionId<br/>retain 3d</i>"]]

  t1 --> p1["Normalise<br/><i>stateless</i>"]
  p1 --> t3[["orders.clean<br/><i>12 parts · key: orderId<br/>compacted</i>"]]
  t3 --> p2["Enrich + join<br/><i>stateful · 30 min window</i>"]
  t2 --> p2
  p2 --> st[("RocksDB state<br/><i>checkpoint 60s</i>")]
  p2 --> t4[["orders.enriched<br/><i>12 parts · retain 30d</i>"]]
  p2 --> dlq[["orders.dlq"]]

  t4 --> sink1["Serving store"]
  t4 --> sink2["Lakehouse sink<br/><i>5 min commit</i>"]

What it is

The streaming pipeline with the decisions that are expensive to change written on it: partition count, partition key, retention, compaction, window size and checkpoint interval. Every one of those is either a correctness property or a recovery-time property, and none of them are visible in code review.

When you produce it

When designing a streaming pipeline and before the first topic is created — repartitioning a live topic with committed consumer offsets is a migration. Revisit whenever consumer lag becomes an incident.

Who reads it

Data and platform engineers who own it. Operations, who need to know which stages hold state and therefore which are slow to restart. Anyone planning a backfill.

What good looks like

Partition key on every topic. It determines ordering, and ordering is usually the requirement someone assumed.
Retention on every topic, checked against the longest replay you might need.
Stateful stages marked, with state size and checkpoint interval — those give the restore time.
The dead letter path exists and has an owner.
Compacted topics distinguished from retained ones; they are different things.

Common mistakes

Retention shorter than the backfill window, discovered on the day you need to reprocess.
Repartitioning silently in the middle of a topology, losing the ordering guarantee the consumer assumed.
No dead letter queue, so one poison message stops a partition.
Ignoring state size, then finding recovery takes forty minutes.