Load Shedding
Deliberately rejecting a portion of incoming work during overload so that the remainder can be served correctly.
Under overload, a system that accepts everything serves nothing: queues grow, latency exceeds every client's timeout, and all capacity goes to producing responses nobody is waiting for. Shedding is the choice to fail some requests fast so the rest succeed.
Three design decisions make it effective rather than arbitrary:
What to shed. Priority should be agreed with the business in advance — checkout and payment protected, recommendations and personalisation shed first. Shedding uniformly is better than collapsing, but shedding by value is much better than either.
When to shed. Queue depth or concurrency is a better trigger than CPU, because a service can be latency-bound while CPU is idle. Shed before saturation, not after.
Which requests to drop. Counter-intuitively, prefer to drop requests that have already spent most of their deadline — completing them helps nobody, since the client has stopped waiting. Propagating a deadline with the request is what makes this possible.
Shed with a clear signal — HTTP 429 or 503 with Retry-After — so well-behaved clients back off
rather than immediately retrying and deepening the overload.