Model Routing and Cascades
Why sending every request to the strongest model is usually wrong, the two routing patterns and what each requires, and the economics that decide whether routing pays.
Request difficulty is heavily skewed. Most requests to a production LLM application are simple, and a small fraction are hard. Sending all of them to the model that can handle the hard ones means paying frontier prices for the easy majority, and the gap between the cheapest adequate model and the strongest available one is frequently more than an order of magnitude in cost per token.
Two patterns
Routing decides up front which model handles a request, based on the input alone. It needs a classifier, which is itself a model, so its cost and its error rate enter the calculation. A misroute sends a hard request to a weak model and produces a bad answer with no signal that anything went wrong, which makes routing errors expensive in a way that is invisible in the aggregate cost saving.
Cascading tries the cheap model first, evaluates the result, and escalates if the result is inadequate. It needs no difficulty classifier and instead needs a quality signal on the output, which is a different and often easier problem: the cheap model's confidence, a schema validation failure, a verifier's judgement, or a self-consistency check across samples.
The tradeoff between them is latency against accuracy. A cascade that escalates pays the cheap model's latency plus the expensive model's, so the escalated path is slower than routing directly would have been. Routing pays a small classifier cost on every request and gets the decision wrong sometimes.
The economics
Routing pays when the cheap model handles a large share of traffic adequately and the price gap is large. With 80 percent of requests handled by a model at a tenth the cost, the blended cost falls to roughly 28 percent of the baseline, which is a large saving that survives a substantial misroute rate.
It stops paying when the difficulty distribution is flat, when the price gap is small, or when the quality cost of a misroute is high. A system where a wrong answer is expensive should route conservatively or cascade, since a cascade's failure mode is extra latency rather than a bad answer.
When it breaks
The router needs its own evaluation. Routing accuracy is a distinct metric from end-to-end quality, and a router that is 90 percent accurate on a distribution where 20 percent of requests are hard is misrouting half the hard ones. Reporting only blended cost hides this entirely.
Difficulty is not observable from the input. For many tasks, whether a request is hard depends on facts the router cannot see: whether the retrieved context happens to contain the answer, whether the user's phrasing is ambiguous in a way that only matters here. Cascading sidesteps this because it judges the output rather than predicting the difficulty.
Escalation criteria drift. A cascade's threshold is tuned against a model version, and both models change. Without monitoring the escalation rate, a model update can silently move it, changing both cost and quality with no deployment.
Multiple models multiply the surface. Two models means two sets of prompts, two evaluation runs, two sets of failure modes and two deprecation schedules. The operational cost is real and is usually omitted from the saving calculation.
12 flashcards for this concept
Click a card to reveal the answer.