A streaming pipeline's cost grows faster than its event volume. What are the drivers?

The drivers

1. State growth. Stateful operators holding entries for every key ever seen, with no time-to-live. State drives memory, checkpoint size, checkpoint duration and recovery time — and it grows with cumulative history rather than with throughput.

2. Over-parallelisation. Partition and task counts set generously and never revisited. Each carries fixed overhead, and beyond the point where parallelism helps, more tasks cost more and process no faster.

3. Retention set long "just in case". Stream retention is storage, and long retention across many topics is a substantial and easily-forgotten line item.

4. Windowing multiplying cardinality. A sliding window over a high-cardinality dimension produces many window instances per key, and state grows accordingly.

5. Replication and durability settings applied uniformly, including to streams where the data is regenerable.

6. Always-on capacity for a workload with a diurnal pattern, sized for peak and running continuously.

7. Backfills competing with live processing, causing over-provisioning of the live path to absorb them.

The controls

• State time-to-live, which is the single largest lever for a stateful pipeline.
• Retention by topic, matched to the actual replay requirement rather than set uniformly.
• Right-size parallelism against measured throughput, revisited as volume changes.
• Aggregate early, before the expensive stages, so downstream operators handle less.
• Filter at the source, since the cheapest event is the one never published.
• Isolate backfills, so the live path is sized for live traffic.
• Cost attribution per pipeline and per team, without which optimisation targets the wrong thing.

The reframing

Cost per useful output, not per event processed. A pipeline computing features nobody consumes is entirely waste regardless of its efficiency — and the highest-return question is usually which outputs are still consumed, not how to compute them more cheaply.