Flipkart Marketplace Platform  ·  View 21 of 25  ·  6 · Operations

Observability and Operations

How anyone knows it is working, and who gets woken up when it is not.

Editable source SVG draw.io All views
Emit Collect Store Consume Act Metrics OpenTelemetry SDK OTel Collector per node Prometheus + Thanos 13 mo Grafana SLO boards Page on burn rate multi-window Logs Structured JSON trace id bound Fluent Bit Loki 30 d hot Incident search Runbook link Traces OTel auto-instr 1% + tail sample OTel Collector Tempo 7 d Checkout latency Blame the hop service map Business Order and pay events Flink aggregates 10 s window Pinot GMV, conversion War room board sale day Page on conversion drop vs baseline Pipeline health Consumer lag Kafka exporter Prometheus Lag and DLQ board Auto scale consumers KEDA Cost Namespace tags OpenCost per service Cost mart Iceberg Unit cost per order Observability — Signal to Action Application we own Interface / broker Data store Security / platform Queue / topic The empty cell is deliberate: cost has no automated action, it has a monthly review. Business signal is treated as a first-class alerting source because conversion drops before CPU does. v 1.0 · owner SRE · date 2026-09

Decisions

  • OpenTelemetry everywhere, then an open-source backend per signal — Prometheus and Thanos, Loki, Tempo, Grafana
  • Business signal is a first-class alerting source: conversion and order rate drop before CPU does
  • Paging is on SLO burn rate over multiple windows, not on thresholds, so a slow leak and a cliff both page correctly

What is tracked

  • Latency, error rate and traffic per service; consumer lag and dead-letter depth per topic
  • Checkout conversion, payment success rate by instrument and by PSP, oversell count, inventory drift
  • Unit cost per order, reviewed monthly rather than alerted

Assumptions

  • Tail-based trace sampling keeps every error and slow trace while sampling 1% of the rest
  • 13 months of metric retention supports year-on-year sale comparisons, which is the actual requirement
  • The empty cell on cost is deliberate: there is no automated action, and pretending otherwise would be decoration
◀  Delivery Pipeline and Environments All views Peak Load and Graceful Degradation  ▶
Solution Architecture v1.0 · Open source on Microsoft Azure · Enterprise Architecture · 2026-0921-observability