Fallbacks, Timeouts and Degradation
How to build an LLM feature that survives a provider outage, why the usual retry patterns need adjusting for generation, and what degrading gracefully looks like when the core capability is unavailable.
An application that depends on a hosted model has taken a dependency on a third party with its own incidents, rate limits and capacity constraints. Standard resilience patterns apply and several need adapting, because generation has properties that ordinary API calls do not: high and variable latency, partial results, and cost per attempt.
Timeouts
A generation request's duration depends on output length, so a single fixed timeout is wrong for both short and long responses. The useful pattern is a time-to-first-token timeout, which catches queueing and capacity problems quickly, plus an overall deadline sized to the expected output.
Streaming complicates cancellation: a request abandoned after partial output has still consumed tokens and still incurs cost, and the partial output may be usable. Deciding whether to keep, discard or complete partial results is an application-level choice that has to be made deliberately.
Retries
Retrying a generation is not idempotent in the usual sense, since a retry produces different output rather than the same result. That is sometimes exactly what is wanted, when the first attempt failed validation, and sometimes wasteful, when the failure was a timeout on a request that was about to succeed.
Retries must be budgeted, because each costs tokens. An unbounded retry loop on a request the model consistently fails is an unbounded bill, which is why retry counts belong in the spend guardrails rather than only in the resilience configuration. Exponential backoff with jitter applies as usual, and rate-limit responses should be honoured rather than retried immediately.
Fallbacks
A different provider for the same capability gives real independence and requires prompts that work on both, which in practice means testing against both continuously rather than discovering the difference during an incident.
A smaller or self-hosted model provides degraded capability that is under your control, which is often more valuable during an incident than equivalent capability that shares the failure.
A cached response for a repeated or similar request, which is exact for identical inputs and requires a similarity threshold otherwise.
A non-AI path: a rules-based response, a template, a search result, or telling the user the feature is unavailable. Explicit unavailability is a legitimate design and is better than a confidently wrong answer produced by a fallback that was not up to the task.
When it breaks
Fallbacks are untested and fail when used. A path exercised only during incidents has bit-rotted since it was written. Routing a small share of traffic through it continuously is what keeps it working.
Degraded quality is invisible. A fallback that returns worse answers without any signal means users experience a quality drop nobody measured. The fallback path should be logged and its quality monitored separately.
Correlated failure defeats the design. Two providers on the same cloud region, or a fallback model served by the same infrastructure, fail together. Independence has to be checked rather than assumed from the vendor names.
Cost during an incident is unbudgeted. Retries plus fallbacks plus a surge of user retries can multiply spend precisely when the system is degraded. Rate limiting during degradation protects both the budget and the recovering dependency.
12 flashcards for this concept
Click a card to reveal the answer.