Backpressure in Streaming
A slow downstream stage signalling upstream to slow down, so that queues stay bounded instead of consuming memory until the job dies.
A pipeline is only as fast as its slowest stage. Without a feedback mechanism, faster upstream stages keep producing into buffers that grow until memory is exhausted, and the failure appears as an out-of-memory kill in a component that was working correctly.
Pull-based systems get a degree of this for free: a consumer reads at its own rate from a durable log, and falling behind manifests as lag rather than as memory pressure — which is one of the underappreciated architectural advantages of putting a log between stages instead of connecting processors directly.
Within a processing topology, frameworks propagate backpressure across stages so the source slows to the rate the slowest operator can sustain. The important operational point is that backpressure is a symptom to be diagnosed, not a mechanism to be relied upon. A job running under sustained backpressure is not healthy; it is failing to keep up, and the lag is growing somewhere upstream.
Where it becomes dangerous is at the boundary with systems that cannot be slowed: a device fleet, a payment network, a partner feed. Those need a durable buffer at the edge and an explicit policy for what happens when it fills — shed load, sample, degrade the computation, or spill to storage — chosen deliberately rather than discovered when the buffer fills at three in the morning.