A platform must choose which model serves which request class. What should drive the decision, and what changes over time?

What should drive it

Task-specific evaluation on your own data, plus cost and latency per request class. Public benchmarks correlate imperfectly with any particular application, and a model that scores well generally may be worse for your specific task, corpus and prompt.

The decision is per request class rather than global:

• High-stakes, low-volume interactions — where an error is expensive — justify the most capable and most expensive option.
• High-volume, simple interactions — classification, extraction, routing, short summarisation — are frequently served well by a smaller, faster, cheaper model, and this is where most of the cost is.
• Latency-critical interactions may require a smaller model regardless of quality preference.
• Interactions with strict output structure may be better served by a smaller model plus validation and retry than by a larger one.

The architectural consequence

Model choice must be configuration, not code. Two properties follow, and both are structural:

1. A routing layer that selects the model per request class, so the choice can change without a deployment, and so A/B comparison between models on real traffic is possible.

2. An abstraction over providers and models thin enough not to become a lowest common denominator, but present — because the decision will change, repeatedly, and a codebase with model choices scattered through it cannot follow.

What changes over time

Everything. New models appear, prices change, capabilities shift, and a task that required the most capable option becomes servable by something cheaper.

That makes the ability to re-evaluate cheaply the durable capability, not the current choice:

• A held-out evaluation set of real tasks with known-good outputs, maintained as the product evolves.
• A comparison harness that runs a candidate model against it and reports differences.
• Traffic-based comparison for the final check, since offline evaluation correlates imperfectly with user outcomes.
• Cost and latency measured per request class, so the trade is quantified rather than argued.

The failure to avoid

Selecting once and never revisiting, so the platform runs an expensive model for a task a cheaper one now handles — or a weaker one for a task where quality has become the constraint.

Without an evaluation harness this cannot be detected, which is why evaluation infrastructure is the prerequisite for model selection being an ongoing decision rather than a historical one.