intermediate 2 min answer

A quick-commerce platform must decide whether a data flow should be streaming or batch. What actually decides it?

zeptostreamingbatchlatencycost
Show the full answer Hide the answer

What decides it

Whether a decision is made on the data before the next batch would have run.

  • Streaming when the data drives an operational decision within its production interval: stock availability during a checkout, courier assignment, demand signals that change dispatch.
  • Batch when the consumer is a human looking at a report, a model being trained, or a reconciliation — none of which is improved by the data arriving four hours earlier.

Latency requirement is the decider, and it is frequently asserted rather than derived. "We want it in real time" should be answered with "what decision changes if it arrives in five minutes instead of five hours", and the answer is often nothing.

What streaming costs

  • Continuous operation rather than a scheduled job, so a failure is an incident rather than a rerun.
  • Harder correctness: out-of-order events, late data, watermarks, exactly-once semantics — all of which batch avoids by processing a complete bounded set.
  • Harder reprocessing. A batch job is re-run; a streaming pipeline requires replay from a retained log, which must have been retained.
  • More expensive per record at low volumes, since the infrastructure runs continuously.

The middle option that is under-used

Micro-batching: frequent small batches, minutes apart. It gives most of the freshness with most of batch's simplicity — bounded sets, easy reprocessing, simpler correctness — and it is sufficient for a large proportion of requirements that are stated as streaming.

The quick-commerce specific split

Stock reservation is transactional and neither streaming nor batch — it is an operational write requiring strong consistency at the point of commitment.

Stock projections for browse are streaming, since staleness of seconds is acceptable and freshness matters. Demand forecasting and stock placement are batch, running hourly or daily against complete data.

Applying one model to all three is what produces either an unaffordable pipeline or a wrong promise — and the classification is a product question rather than a technical one.