A team splits a single batch job into seven stages connected by queues, so each stage can scale and be retried independently. Throughput improves. What have they given up, and when does that bill arrive?

What is gained, quantified

Independent scaling of the expensive stage, so the stage that took 80% of wall-clock time no longer forces the other six to be provisioned for it. Independent retry, so a transient failure in enrichment does not re-run validation. Independent deployment, and a natural place to apply backpressure. For a job whose stages have genuinely different resource profiles, the saving is real and often several times the cost of the original box.

What is paid

End-to-end latency becomes a sum of queue waits, not a sum of processing times. Seven hops each with a modest queue means the p99 of the whole is the sum of seven p99s, and queue waits are the least predictable part. A pipeline whose stages each take 50 ms can easily have a p99 of several seconds under load, because one slow stage backs up and every downstream wait inherits it.

Failure semantics multiply. Each queue is at-least-once, so every stage needs idempotency. Seven stages means seven places a message can be duplicated, poisoned or dead-lettered, and seven dead-letter queues that somebody must own. The most common production failure in a pipeline like this is a dead-letter queue nobody reads.

Debugging needs infrastructure that did not exist before. In one process, a stack trace answers "where did record 41229 fail". Across seven stages, that answer requires a correlation id threaded through every message and a way to query by it. If that was not built on day one, it gets built during the first incident.

Operational surface goes from one deployable to seven, with seven sets of alerts, dashboards and scaling policies, and a team that now spends time on the pipeline rather than on what the pipeline computes.

When the bill arrives

Not at build time; the pipeline looks elegant then. It arrives:

• At the first partial failure, when three stages have processed a record and four have not, and somebody must decide what the system's state actually is.
• When the business asks for end-to-end latency, and the answer is "somewhere between 400 ms and 40 seconds depending on queue depth".
• When a schema changes, because each queue is a contract, and seven contracts must be evolved in a compatible order.

How to keep the option to reverse

Keep stages as library functions with thin queue adapters, so a stage can be inlined back into its neighbour by changing wiring rather than rewriting logic. Split on a measured bottleneck, not on a diagram. The right number of stages is the number of genuinely different resource profiles in the work, which for most jobs is two or three, not seven.

When this is the wrong answer entirely

If all seven stages are CPU-bound at similar rates and the job fits comfortably in its window, the pipeline buys nothing and costs the whole list above. The decisive question is whether any stage has a resource profile or failure rate different enough to justify its own scaling policy. One slow stage justifies one split, not six.