An e-commerce platform designs explicit degradation modes for extreme demand events. What must be true of the mode ladder for it to work on the day?
Show the full answer Hide the answer
What must be true
1. The ordering is a business decision, made in advance. Which capability is expendable, and in what sequence, is a revenue and trust judgement — not a technical one. Recommendations before reviews, reviews before search facets, facets before checkout. An engineer's implicit judgement, encoded in code and discovered by the business afterwards, is an organisational failure regardless of how good the code is.
2. Transitions are automatic, triggered by measured signals. At the peak of a synchronised demand event, the window between "load is rising" and "unrecoverable" can be under a minute. Any transition requiring a human decision is an aspiration. Triggers are queue depth, latency percentiles, dependency error rates and concurrency utilisation.
3. Every rung has been exercised under realistic load. The characteristic failure is that shedding recommendations also breaks the homepage, because the fallback path was never run. A degradation ladder that has not been rehearsed is not a control.
4. Fallbacks require no dependencies. A static bestseller list beats personalised recommendations that need a service which is, by hypothesis, already struggling. If the fallback shares a dependency with the primary, it is not a fallback.
5. Shedding is by request class, not uniform. Anonymous browsing before authenticated sessions; new sessions before in-flight checkouts; enrichment before core. Uniform shedding wastes the ability to protect what matters.
6. Recovery is automatic with hysteresis, so the system does not oscillate between rungs — and does not require a deployment to return to full service, or the degradation outlives the event.
7. The current mode is visible. Operators must see which rung the system is on and why, and be able to force a rung up or down without a deploy.
What is never on the ladder
Inventory correctness. Overselling is a business failure that degradation must not cause, and it cannot be compensated — a double charge is refundable, a double sale of a limited item is not.
So the reservation path is protected by shedding everything around it rather than by weakening it. That asymmetry is the core design insight: identify the one invariant that cannot be compensated, protect it properly, and let everything else degrade.
The failure to avoid
A degradation capability that exists in code, has never been triggered, and is used for the first time under peak load. It will not work, and it will fail in a way that makes the incident worse — which is why the rehearsal requirement is not optional advice but the condition that makes the rest meaningful.