A recommendation service's model server is down at peak. Should the application fall back to an older model, precomputed candidates, popularity baselines or cached per-user results — and how is the fallback tested?

The layered fallback

Order by quality retained, with each layer cheaper and more robust than the one above:

1. Cached per-user results from recent successful inference. Highest quality retained — it is that user's genuine personalisation, merely stale — and it works only for users who have been served recently, which at peak is most of them.
2. Precomputed candidates, generated offline in batch and stored per user. Personalised, hours stale, and available for every user rather than only recently-active ones. This is the most valuable layer to build because its coverage is complete.
3. An older or smaller model served from a separate pool. Useful if the outage is specific to the current model or its hardware, and useless if the cause is shared infrastructure — which is the more common case, making this layer less valuable than it appears.
4. Popularity or trending baselines, segmented by whatever coarse attributes are cheaply available — locale, category, device. Not personalised, entirely robust, and typically retains more engagement than teams expect, which is a useful and slightly deflating fact.
5. A static curated list. Always works, requires no infrastructure, and is the floor.

Never fail the page. A recommendation module is an enhancement; the page must render without it, with the section omitted or replaced. Coupling page rendering to a model server is the actual defect in most of these incidents.

Testing the fallback, which is where this usually fails

A fallback that has never served production load is an untested code path that will run under the worst possible conditions.

• Exercise it in production regularly — a scheduled window, or a small percentage of traffic permanently served by each layer. A fallback not used in six months does not work, because the code beneath it has changed.
• Load test each layer at full traffic. The popularity baseline reading from a database that has never served the full request volume will fail exactly when it is needed. Precompute and cache the baseline aggressively, since it is identical for large groups of users.
• Verify the trigger, not only the behaviour — a circuit breaker keyed on errors will not open when the model server is slow rather than failing, which is the more common mode.
• Measure the business impact of each layer, in a deliberate experiment. This number decides how much to invest in the upper layers, and teams that have it usually find the gap between layer 2 and layer 4 is smaller than assumed.
• Check the fallback's own dependencies. A cache fallback whose cache shares infrastructure with the failing model server is not a fallback.

The design principle

The fallback path must be simpler and have strictly fewer dependencies than the primary path. A fallback that requires the same database, the same service mesh and the same feature store has not reduced the number of things that must work — it has only changed which model runs, which is rarely the thing that failed.