Review this ingest path for a live-sport streaming platform of Hotstar's kind - client events to Kafka with 12 partitions, a Flink job, Redis, a second Kafka topic, a Postgres sink, plus a transactional outbox. Peak is 2,000 events per second and the only consumer is an hourly dashboard. What would you remove?
Show the full answer Hide the answer
What is actually required
2,000 events per second is 7.2 million an hour and roughly 170 million a day. At a few hundred bytes each that is tens of gigabytes a day, and a batched insert into one Postgres table on a mid-sized instance absorbs it with room to spare. The consumer is hourly, so there is no latency requirement anywhere in this pipeline - nothing downstream can tell the difference between a one-second and a fifty-minute path.
The real requirement hiding in the description is burst tolerance, not throughput. A live-sport audience does not arrive smoothly: concurrency on this kind of platform can rise by an order of magnitude within a minute of a wicket or a penalty, and the ingest path has to not fall over when it does.
What I would remove, and why it is safe
- Flink. It is carrying no windowed state, no joins and no event-time logic. It brings a checkpoint store, a state backend, savepoint-compatible upgrades and a second scheduling system to operate - all for a pass-through.
- The Redis hop. Nothing reads it. It is a cache in a pipeline with no cache consumer, and it is now a component whose loss stops ingestion.
- The second Kafka topic. A topic exists to decouple a producer from consumers. There are no consumers of this one.
- The outbox. An outbox makes a database write and a message publish atomic. There is no database write to be atomic with: the Postgres sink is the end of the path, not a participant.
What is left: client events to Kafka, a consumer that batches into Postgres, and an hourly query. Four moving parts become two.
The one change that matters
Make the drop policy explicit at the edge. Bound the client-side buffer, decide in advance which events are shed when the burst exceeds the ingest rate, and record the shed count as a first-class metric. An analytics event lost during the goal is worth far less than a queue that grows until it takes the path down, and the choice should be a line of configuration rather than whatever the buffer happens to do under pressure. Alert on age of the oldest unconsumed record, not on queue length, because length tells you nothing about whether you are falling behind.
What I would leave, even though it looks odd
Kafka itself, and the 12 partitions. Kafka is the replay buffer that lets the hourly job be re-run after a bad deploy, and that is worth one component. 12 partitions for 2,000 events per second is more than needed and costs nothing - and partitions are awkward to reduce later, so over-provisioning them is the cheap direction of the mistake.
When this is the right architecture
Name the trigger that would justify re-adding what you removed, so the reviewers hear a condition and not a preference: a second consumer with a sub-minute requirement (then the second topic earns its place), or windowed aggregation across a session (then a stream processor does, because doing it by hand in a consumer is worse). Until one of those exists, each component is an upgrade to perform, a page to answer and a failure mode to learn.
How I would argue this in the review
Not on cost - the infrastructure bill is not the point. On on-call surface. Every component is something that can be the reason the dashboard is empty on Monday, and an on-call engineer must hold all of it. The team is not being asked to be less ambitious; they are being asked to spend their complexity budget on the burst behaviour, which is the part that will actually hurt.