Request Coalescing

When a popular cached object expires or is evicted, every concurrent request for it misses at the same moment. Without coalescing, each miss becomes an independent upstream request, and the origin receives a burst proportional to the object's popularity — precisely for the objects it can least afford.

Coalescing keeps an in-flight map keyed by resource identity. The first miss issues the upstream fetch; subsequent misses for the same key attach to the in-flight request and receive the same result. N concurrent misses produce one upstream call.

Why it matters

It converts the origin's load from proportional to request rate into proportional to the number of distinct objects being refreshed. That is a fundamentally different scaling relationship, and it is what allows a small origin to sit behind an enormous edge fleet.

Implementation patterns

• Single-flight per cache node, the basic building block.
• Origin shielding, which is coalescing across a tier: hundreds of edge nodes route their misses through a small set of parent caches, so coalescing happens hierarchically rather than only within each node. Without it, per-node coalescing still lets hundreds of nodes each make one request.
• Stale-while-revalidate, which sidesteps the problem for most cases: serve the stale copy immediately and refresh in the background, so no request ever waits on a miss.
• Soft and hard TTLs, where a single request is elected to refresh at the soft TTL while others continue on the stale value.
• Negative caching — coalescing must apply to misses and errors too, or a nonexistent key becomes an unlimited origin load.
• A bound on wait time. Waiters need their own deadline; if the elected fetch hangs, everyone waiting on it hangs.

Industry example

The viral-video case makes the arithmetic vivid. One object becomes extremely popular within minutes. Edge nodes worldwide either do not have it yet or see it expire around the same time. With per-node coalescing alone, a thousand edge nodes generate a thousand origin requests — better than a million, but still a burst the origin was not sized for.

Adding origin shielding reduces that to a handful. Adding stale-while-revalidate means that after the first fetch, no user ever waits for a refresh again. Adding proactive cache warming for predictable events — a scheduled premiere, a product launch, an expected news moment — means the object is in place before the demand exists.

Those four mechanisms together are why a single origin can survive a global viral event, and any one of them alone is insufficient.

Failure scenarios

• Coalescing only within a node, leaving fleet-wide amplification untouched.
• No negative caching, so requests for a missing object bypass the cache entirely and hammer origin.
• Unbounded waiters, converting one slow upstream fetch into a thread-exhaustion event across the cache tier.
• Synchronised expiry. Every replica's copy expiring at the same instant recreates the stampede on schedule. Add jitter to TTLs.
• Coalescing personalised responses. If the cache key omits a dimension that varies per user, one user's response is served to everyone — the most severe correctness bug in this area, and a genuine security incident when the dimension is identity.

Trade-offs

Waiters experience the latency of the single upstream fetch rather than issuing their own, which is usually faster than a stampeded origin but is a real coupling: one slow fetch delays many requests. The in-flight map is shared mutable state on the hot path. And correctness now depends entirely on the cache key being complete, which is easy to get wrong and catastrophic when wrong.

Interview question

"A single video goes viral and receives millions of requests in minutes. Explain how CDN caching, origin shielding, request collapsing, stale-while-revalidate and cache warming interact — and tell me which one you would implement first if you could only have one."