Pipes and Filters
Independent transformation steps composed into a pipeline.
4 to work through
-
intermediate
A logistics platform processes tracking events through validation, enrichment, deduplication, state transition and fan-out. What does the pipes-and-filters shape buy, and where does it break down?
2 min answer -
intermediate
A nightly recommendation pipeline has one stage consuming 80% of the wall-clock time. How do you diagnose it and what are your options?
3 min answer -
intermediate
A team splits a single batch job into seven stages connected by queues, so each stage can scale and be retried independently. Throughput improves. What have they given up, and when does that bill arrive?
2 min answer -
intermediate
A video platform's transcoding pipeline is a sequence of independent processing stages. What does the pipes-and-filters structure buy, and what must be added for it to work at scale?
2 min answer
3 terms in this topic
Filter Composability
The property that makes a pipeline style work — each stage transforms input to output with no shared state, so stages can be reordered, reused and sc…
patternPipes and Filters
Decomposing processing into independent stages connected by data channels, so each stage can be developed, tested, scaled and restarted alone.
conceptSelective Reliability
Retransmitting only what is still useful and discarding what has expired - because for time-sensitive data, late delivery consumes capacity without p…
Neighbouring topics
Architecture Patterns
General material on architectural patterns and their trade-offs.
Layered Architecture
The default shape, its clarity, and where a technical partition fails.
Event-Driven Architecture
Publishing facts, and trading comprehensibility for decoupling.
Publish/Subscribe
Broadcast to every subscriber, as distinct from work distribution.
Competing Consumers
Scaling throughput with instances, at the cost of ordering.
Saga
Local transactions with compensating actions across services.
CQRS
Separate models for writing and reading, each optimised for its job.
Event Sourcing
The event log as the system of record, with state as a projection.
Outbox
Making the event atomic with the business write it describes.
Strangler Fig
Incremental replacement behind a routing facade.
Anti-Corruption Layer
Translating a foreign model at the boundary so it does not leak in.
Sidecar & Ambassador
Cross-cutting behaviour in a co-deployed process.
Service Mesh
Sidecars applied estate-wide, and the scale at which that pays.
API Gateway
A single entry point for policy, routing and protocol translation.
Backend for Frontend
A narrow backend per client experience, owned by that client's team.
Cell-Based Architecture
Complete isolated copies each serving a subset of customers.
Sharding Patterns
Directory versus embedded keys, logical shards and rebalancing.
Materialized Views
Precomputed query results, refreshed incrementally or in full.
Orchestration vs Choreography
A coordinator that knows the flow, or services that react to events.