Degradation Ladder

Every overloaded system sheds something. The only question is whether the choice was made deliberately in a room, or accidentally by whatever component happens to fail first — and the accidental answer is frequently the most complex, most dependency-laden path in the system, which is usually checkout, payment or whatever else matters most.

A degradation ladder makes the choice in advance: a specific ordered list, from least valuable to most, each rung individually switchable at runtime.

Why it matters

It converts "how much traffic can we survive?" from an unknown into a measured number, because the system's capacity in degraded mode can be load tested and is typically far higher than its capacity in full mode. That multiple is the entire value proposition, and it is available only if the switches exist and have been exercised.

It also moves a commercial decision out of the incident. Whether to disable recommendations to protect checkout is a business judgement; making it at 2am from ambiguous signals, under pressure, without the business present, produces worse decisions than making it in advance with everyone in the room.

Implementation patterns

• Order the ladder from the critical user journey outward. Anything not on the journey is a candidate; anything on it degrades last.
• A runtime switch per rung that requires no deployment — and crucially one that does not depend on the control plane of the failing system. A kill switch that needs a healthy configuration service is unavailable during the incidents that matter; local caching with a long stale-serving window is the fix.
• A named owner and a pre-agreed trigger condition per rung, so the decision is authorised rather than debated.
• A defined, tested fallback behaviour — "recommendations off" must mean a popularity list, a cached response, or a section that renders correctly when empty. The fallback path must itself be load tested, because it will serve full traffic for the first time under the worst conditions.
• Measure the capacity each rung releases, so the ladder is ordered by benefit rather than intuition — teams routinely discover the rung they assumed was most valuable frees almost nothing.
• A user-facing message where the absence is noticeable, since communicated degradation is tolerated and silent degradation reads as a broken product.
• Scheduled exercise of every switch in production. A switch not thrown in six months does not work, because the code beneath it has changed.

Industry example

Organisations with scheduled peaks build these as standard practice: streaming platforms preparing for record live-event concurrency, retailers preparing for annual sale events, payment processors preparing for peak commerce days. The published accounts consistently describe a pre-agreed list of non-essential features with owners and trigger points, rehearsed before the event — chat, statistics overlays, personalised rails, recommendations, and analytics being the usual first rungs.

The recurring lesson from those accounts is that the rehearsal, not the list, is what makes it work. Ladders that exist only as documents fail at the moment of use, because a switch that has never been thrown in production is an untested code path in the most stressful conditions imaginable.

Failure scenarios

• Switches requiring a deploy, which makes them unusable at the moment they are needed.
• A kill switch dependent on the failing configuration service.
• Untested fallbacks, where turning off the primary path reveals that the fallback also fails, or cannot handle the load.
• No owner or trigger, so the ladder exists and nobody is willing to pull it.
• Rungs ordered by intuition, releasing negligible capacity while costing real product value.
• Silent degradation, generating support load and reputational damage disproportionate to the outage.
• Switches that decay because nothing exercises them.
• Degrading the wrong direction under automation — an automated system that sheds by error rate may shed the critical path first, because it is the one under most stress.

Trade-offs

Building and maintaining a ladder is ongoing work that produces no visible value on ordinary days, which is why it is consistently under-invested in. Each rung is a code path that must be maintained, tested and exercised, and the switches themselves add configuration surface and a class of incident caused by someone flipping one by mistake.

There is also a product cost to designing for absence: features must be built so they can be removed cleanly, which occasionally constrains how they are integrated and adds defensive handling throughout the UI.

The trade is continuous maintenance effort and some design constraint in exchange for a much higher survivable traffic ceiling and a controlled rather than arbitrary failure mode. For a system with scheduled peaks or unpredictable spikes it is close to mandatory; for a steady internal system with generous headroom it is machinery that will never be used.

Interview question

"Traffic is 4× forecast and climbing. Tell me the first three things you turn off, who authorises each, how long it takes, and how you know the fallback can handle the load — then tell me when you last actually did it."