Flipkart Marketplace Platform  ·  View 11 of 25  ·  4 · Data

Data Flow

Where data comes from, what happens on the way, and who is allowed to consume it.

Editable source SVG draw.io All views
Sources App clickstream 4B events/day Domain events 38 topics OLTP change feed Debezium Partner feeds carriers, PSPs Ingest Kafka 7 d hot, tiered Schema registry Avro · compat gate Dead letter topics replayable Process Flink streaming exactly once Spark batch nightly restate Quality gates expectations Store Iceberg lakehouse bronze/silver/gold Pinot sub-second OLAP Feature store Feast online/offline Serve Trino ad hoc SQL Superset dashboards Metrics API seller and internal Model serving KServe Consume Sellers Analysts Ranking and fraud Finance and audit poison messages online features training sets Data Flow — Source to Decision Application we own External / third party Queue / topic Security / platform Data store Interface / broker Person or role failure / alternate event / async batch One log, two speeds. Flink computes what has to be right within seconds; Spark restates the same tables nightly so a late or corrected event never leaves the marts wrong. v 1.0 · owner Data Architecture · date 2026-09

Decisions

  • One log, two speeds: Flink computes what must be right in seconds, Spark restates the same tables nightly
  • Schema registry compatibility is a merge gate, so a producer cannot break a consumer by deploying
  • Poison messages go to a dead-letter topic and are replayable; nothing is dropped silently

Numbers

  • About 4B clickstream events a day, plus 38 domain topics
  • Kafka 7-day hot retention with tiered storage to Blob beyond that
  • Streaming freshness target 60 seconds for seller analytics, 30 seconds for search

Risks

  • Late and corrected events are the standard cause of a wrong mart — the nightly restatement exists for this and must not be optimised away
  • Clickstream volume is the dominant lakehouse cost driver; sampling policy is reviewed monthly
  • Feature-store skew between training and serving is the failure mode nobody notices until conversion drops
◀  Data Architecture and Ownership All views Event Backbone  ▶
Solution Architecture v1.0 · Open source on Microsoft Azure · Enterprise Architecture · 2026-0911-data-flow