Model Cascade
also called Tiered Inference, Escalation Ladder
A cheap fast model handling the clear majority of cases with escalation to a larger model or a human for the uncertain ones - usually a large cost reduction with no quality loss, because the expensive path runs on a fraction of traffic.
Choosing one model for a task forces a single point on the cost-quality curve, and that point is wrong for most of the traffic: the easy cases are over-served and the hard ones may still be under-served.
A cascade routes by difficulty. A small model answers the cases it is confident about; the rest escalate. For most classification, moderation, extraction and routing tasks the easy cases are the overwhelming majority, so the expensive path carries a small fraction of the volume.
Why it matters
It is the largest single cost lever available in most production model deployments, and unlike prompt optimisation it does not trade quality for savings — the hard cases still receive the expensive treatment.
It also degrades gracefully: when the large model is unavailable or rate-limited, the escalation threshold can be raised, accepting a measured quality reduction rather than failing.
Implementation patterns
- Calibrated confidence, which is a separate engineering problem. A model's raw output probability is frequently a poor confidence estimate, and using it directly escalates the wrong cases.
- Escalate on consequence as well as confidence. A high-stakes item should go to the expensive path regardless of how confident the cheap model was.
- Measure the escalation rate as a first-class metric. A cascade escalating 40% of the time is not delivering the economics its design assumed, and the rate drifts as inputs change.
- Measure cost per successful outcome, not per call. A cheaper model needing three attempts is not cheaper.
- Evaluate each tier separately, so a change to the cheap model's threshold does not silently change the system's quality.
- Include a human tier where appropriate, treating it as a service with a latency distribution and a capacity limit rather than as an escape hatch.
Industry example
Content platforms such as ShareChat run moderation over enormous volumes in many languages, where every item is processed and the cost of a single expensive model per item would be prohibitive. Model quality varies enormously by language, which makes the cascade's thresholds per language rather than global — and that per-language variation is itself an argument for the pattern, since a single model choice is wrong for most languages.
The same structure applies to support triage, fraud screening, document extraction and search relevance.
Failure scenarios
- Uncalibrated confidence, escalating the wrong cases.
- Escalation rate unmonitored, drifting until the economics no longer hold.
- Consequence ignored, so a high-stakes item is decided by the cheap model because it happened to be confident.
- The cheap tier evaluated only in aggregate, hiding that it is systematically wrong on a subgroup.
- No graceful behaviour when the expensive tier is unavailable.
Trade-offs
A cascade is more machinery: two or more models to evaluate and maintain, a routing decision, a confidence estimator, and a more complex failure surface. For low volumes the saving does not justify it.
It also introduces a systematic bias: the cases the cheap model is confidently wrong about never reach the expensive one. Sampling a proportion of confidently-handled cases through the expensive path is the mitigation, and it costs a little of the saving to buy a measurement of the error the design cannot otherwise see.
Interview question
"Your moderation costs are dominated by model spend. Design a cascade, tell me what you would measure to know it is working, and tell me what class of error it would hide from you."