Pipes and Filters

Definition

A stream of data flows through a sequence of filters (transformations) connected by pipes (channels). Each filter knows only its input and output contract, not its neighbours. The pipeline is composed rather than coded.

Why it matters

The pattern buys independent scaling per stage and restartability. A stage that is ten times more expensive than its neighbours gets ten times the parallelism. A stage that fails is re-run from its input rather than restarting the entire job.

It also buys testability of a kind that monolithic processing does not: a filter is a pure function of its input, so it can be tested with a file.

Industry example

Large-scale data processing at Google — the lineage running from MapReduce through Dataflow and the Beam model — is the pattern taken to its conclusion. The important architectural idea in that lineage is that the pipeline is a declarative description separated from the execution engine, so the same logical pipeline can run over a bounded historical dataset or an unbounded live stream.

That separation is what makes reprocessing tractable. When a bug is found in a transformation, the fix is to correct the filter and re-run over historical input, which is only possible because the stages are stateless functions over durable inputs rather than a tangle of in-place mutations.

The second lesson from that lineage is about time. Once a pipeline processes unbounded data, the distinction between event time (when it happened) and processing time (when we saw it) becomes unavoidable, and the pipeline must state explicitly how long it will wait for late data before emitting a result. Every windowed aggregation is a bet about lateness.

Implementation patterns

• Stateless filters wherever possible, so a stage can be retried or parallelised freely.
• Durable pipes — a log, object storage, or a queue — so a stage crash does not lose the data in flight.
• Explicit contracts between stages, versioned, ideally schema-checked.
• Backpressure or a bounded buffer at each pipe, or a slow stage upstream of a fast one consumes all available memory.
• Idempotent output, because a re-run will re-emit.

Failure scenarios

• A stage with hidden state, so a retry produces a different result than the original run.
• Unbounded pipe buffers converting a slow stage into an out-of-memory failure.
• No dead-letter path for records that can never be processed, so one malformed record halts the pipeline.
• Silent partial failure, where a stage drops records it cannot parse and nobody counts them. Every filter should emit counts of input, output and rejected records.

Trade-offs

Composability and independent scaling, paid for with per-stage serialisation overhead, higher end-to-end latency than a single fused pass, and an operational surface with many moving parts. For a transformation that is genuinely one step over a small dataset, a single function is better and a pipeline is ceremony.

Interview question

"Your nightly pipeline has one stage that takes 80% of the wall-clock time. Walk me through how you would find out why, and the options for fixing it."