Degradation Ladder

A degradation ladder is the answer, written down in advance, to "what do we turn off, in what order, when we cannot serve everything?" Each rung names a capability, the signal that triggers shedding it, the user-visible behaviour that results, and how the rung is recovered.

It exists because at the peak of a demand event the window between "load is rising" and "the system is unrecoverable" can be under a minute. Any decision requiring a human in the loop at that timescale is an aspiration, not a control.

Why it matters

Systems without a ladder do not stay up and refuse gracefully. They fail in whichever order their resource exhaustion happens to produce, which is usually the worst possible order — the checkout path starves because a recommendation call is holding its threads, and the feature that generates no revenue takes down the one that does.

Implementation patterns

• Rungs triggered on measured signals — queue depth, latency percentile, dependency error rate, concurrency utilisation — not on a human's read of a dashboard.
• Shed by request class, not uniformly. Anonymous browsing before authenticated sessions; new sessions before in-flight checkouts; enrichment before core.
• Static fallbacks that require no dependencies. A cached bestseller list beats personalised recommendations that need a service which is, by hypothesis, already struggling.
• Automatic recovery with hysteresis, so the system does not oscillate between rungs.
• Rehearsal. Exercise every rung under realistic load before the event.

Industry example

Extreme synchronised-demand events — the flash sales that platforms like Alibaba run at scale, or a major live-stream drop — are the canonical case. Millions of users arrive within seconds for a small set of items. Capacity cannot be added at that timescale, so the only variable is what the system stops doing.

A mature ladder for such an event looks roughly like: personalised recommendations replaced by a static list; reviews and social proof dropped; search facets reduced to a cached subset; non-essential write paths (wishlists, view tracking) queued rather than executed; and finally admission control placing new sessions in a waiting room while in-flight purchases complete.

Note what is never on the ladder: inventory correctness. Overselling is a business failure that degradation must not cause, so the reservation path is protected by shedding everything around it rather than by weakening it.

Failure scenarios

• A ladder that has never been exercised. Shedding recommendations turns out to break the homepage, because the fallback path was never run. Untested degradation is not a control.
• Ordering decided by engineers alone. Which capability is expendable is a revenue and trust judgement. An implicit engineering judgement discovered by the business afterwards is an organisational failure, not a technical one.
• All-or-nothing shedding. No intermediate rungs, so the system goes from full service to a maintenance page.
• No recovery path. Rungs that require a deployment to reverse, so degradation outlives the event.

Trade-offs

A ladder costs real engineering effort for behaviour used a few times a year, plus the ongoing cost of keeping fallbacks working. It also introduces the risk of shedding when the system did not need to. Against that, it converts an uncontrolled failure into a designed one, and it moves the business decision from the middle of an incident to a room where people can think.

Interview question

"Your platform expects a tenfold traffic spike for a two-hour event. Walk me through what you shed and in what order, who decides that ordering, and how you prove on the day before that each rung actually works."