Model Router
Directing each request to a model chosen by the task's difficulty, cost and latency budget, rather than sending everything to the largest model available.
Most workloads have a distribution: a majority of easy requests — classification, extraction, formatting, simple lookups — and a minority of genuinely hard ones. Sending all of them to the frontier model overpays substantially for the majority.
Routing strategies, in increasing sophistication: static by task type, which is simple and captures much of the benefit; cascade, where a small model attempts first and escalates on low confidence or a validation failure; and classifier-based, where a cheap model predicts required capability.
The gains are real on both cost and latency, since smaller models are also faster. The costs are that routing itself adds latency and a failure mode, quality becomes non-uniform in ways that need per-route evaluation, and a cascade that escalates often is more expensive than going straight to the large model.
It belongs in the AI gateway rather than in each application, alongside failover between providers — which is closely related, since the same abstraction that routes by capability can route around an outage.