Small Model Routing
also called Model Cascade, Tiered Inference
Sending each request to the smallest model that can handle it, escalating to a larger one only when needed.
Cost and latency per request vary by an order of magnitude or more across a provider's range, and a large proportion of production traffic in most applications is genuinely easy: classification, extraction, routing, formatting, short factual answers over retrieved text. Sending all of it to the largest available model is the default and is usually the largest avoidable line in an AI budget.
The routing strategies. Static by task, the simplest and often sufficient: classification and extraction to a small model, complex reasoning and synthesis to a large one. Cascade: try the small model, evaluate confidence or validate the output, escalate on failure — which works well when validation is cheap and deterministic, such as schema conformance or a check against retrieved context. Learned routing on query features, which is more sophisticated and adds a component that itself needs evaluation.
The practice that makes the decision empirical rather than assumed: build the evaluation set first, then measure quality per model per task. Teams consistently discover that a smaller model matches the larger one on a substantial share of their traffic, and that the share is larger than anyone predicted.
The engineering caution: escalation logic must have a bound, or a pathological input cascades through every tier and costs more than going to the largest model directly.