A team runs 400 events per second through a pipeline with Kafka Streams for routing, Flink for windowed aggregation and a Spark job for a nightly correction pass, maintained by four engineers. Two of the three have had production incidents this quarter. What would you remove, what would you keep, and what would you leave alone even though it looks odd?

What is actually required

400 events per second is 1.4 million events an hour, which a single process handles on one core with room to spare. State that number out loud before discussing frameworks, because it is the fact that decides the review. Nothing in this pipeline is a scale problem. The requirements are a routing step, a windowed aggregation, and a correction pass, and the question is how many distinct operational surfaces those three steps justify.

Each engine brings its own deployment model, its own failure modes, its own state backend, its own upgrade cycle and its own on-call knowledge. Three engines for four engineers is roughly one engine's worth of operational learning per engineer, and it is the cause of the incidents, not a coincidence alongside them.

What to remove, and why it is safe

• Flink, for this workload. Windowed aggregation at 400 events per second fits in the same library the routing already uses, or in an analytical store that does windowing in SQL over the raw events. What you give up is sophisticated event-time handling with watermarks, which matters when late data is material. Check that first: if late arrivals are under a second and the aggregation is re-computable, the simpler path is correct.
• The Spark job, if the correction can run as a query. A nightly correction over a day of data at this volume is a few tens of millions of rows, which is a warehouse query, not a cluster job. Keep the job only if the correction logic cannot be expressed in the target store.

The one change that matters

Make the aggregation re-computable from the raw events rather than incrementally maintained. At this volume, recomputing a window from scratch is cheap, and it removes the entire class of problems that streaming state creates: checkpoint growth, recovery time, state schema migration. The pipeline becomes "append raw events, derive views on read", which four engineers can hold in their heads.

What to leave alone

The Kafka topic itself, even though 400 events per second does not need a distributed log. It is providing replay, decoupling and a buffer for consumer restarts, all of which the team would have to rebuild if it were replaced with direct calls. Removing the log is the change that looks like simplification and is not.

How to argue this in the review

Not as "these tools are overkill", which invites a defence of the tools. As an operational budget: with four engineers, the team can run one stateful system well. Ask which of the three they would choose to be expert in, and what the other two would have to deliver to justify the other two-thirds of that budget. The flip condition is honest and should be stated: at 40,000 events per second with sub-second freshness requirements, Flink stops being overhead and starts being the reason the thing works.

When this critique is wrong

If the 400 events per second are a pilot for a workload that is already contracted to grow two orders of magnitude within the year, removing the engine now means reinstating it later with migration cost on top. Ask for the growth commitment in writing before recommending removal; "it might grow" is not a reason to keep three engines, and a signed customer is.