Catch-Up Throttling

When a stream consumer falls behind and then recovers, it processes the backlog as fast as it can. Every downstream system it calls now receives traffic at many times the normal rate, from a system that was just unhealthy, at the moment everything else is also recovering.

Catch-up throttling caps the drain rate so that recovery is a controlled ramp rather than a burst.

Why unthrottled recovery is a distinct failure mode

It is the classic [[metastable-failure]] shape: the original trigger has passed, and the system's own recovery behaviour sustains the outage. The backlog burst overloads a downstream service, which slows, which grows the backlog, which means the burst continues longer.

It also compounds with everything else recovering simultaneously — restarted instances restoring state, reconnecting clients, retry queues draining, caches cold. The aggregate load at recovery routinely exceeds peak normal load by a wide margin, and it arrives when the system is least able to absorb it.

Implementation patterns

• A configured maximum processing rate during catch-up, ideally expressed as a multiple of normal throughput rather than an absolute number, so it scales with the system.
• Rate limits and concurrency caps on downstream calls, which is where the damage actually lands.
• Backpressure honoured rather than bypassed — a consumer that ignores downstream slowness during catch-up converts a freshness problem into an availability problem.
• Staggered restarts, so instances do not simultaneously restore state from the same remote storage.
• Jittered reconnection with exponential backoff on the client side, since a [[reconnect-storm]] is the same failure in the connection layer.
• Recovery time reported including catch-up, so the throttle's cost is measured honestly against the incident it prevents.
• Backfills and replays isolated from the live path, on separate capacity, so a deliberate reprocessing cannot starve current processing.

Industry example

Platforms holding very large numbers of persistent connections learn this first and hardest: a gateway restart causes millions of clients to reconnect within seconds, and the reconnection load is far greater than steady-state load. The mitigations that work are server-directed reconnect timing, jitter, and admission control that sheds reconnections rather than accepting all of them and failing everyone.

The same pattern appears in stream processing at recovery, in cache warming after a flush, and in retry queues draining after a dependency returns. In each case the discipline is identical: the recovery path needs its own capacity plan, and it is usually the path nobody load-tested.

Failure scenarios

• Unthrottled drain, overloading a downstream that had nothing to do with the original fault.
• Retry storms stacking on catch-up, doubling the burst.
• Simultaneous state restore, saturating the storage that recovery depends on.
• Throttle set so low that catch-up never completes under sustained load — a throttle must exceed the arrival rate or the backlog grows forever.
• Recovery untested, so the drain rate is discovered during the incident it causes.

Trade-offs

Throttling extends the freshness incident: the system stays behind for longer by design. That is the correct trade when the alternative is a broader availability incident, and the wrong trade when the data has a hard deadline.

The way to make it a decision rather than a guess is to state, per pipeline, whether being late or being down is worse — and to set the throttle from that answer rather than from a default. Most real-time systems answer "late", and are configured as though they had answered "down".

Interview question

"A consumer group falls two hours behind during an incident. The dependency recovers and the consumers start catching up — and ten minutes later the whole platform is worse than it was during the original outage. What happened, and what would you have built to prevent it?"