Multi-Model Strategies: Splitting Work Across Two or Three Models

Most teams pick one model and send everything to it. The model is chosen for the hardest thing in the workload, and then it also handles the trivial things, at the price of the hardest thing.

That is a defensible starting position and an expensive steady state. Real workloads are not uniform in difficulty, and the distribution is usually lopsided: a small fraction of requests genuinely need frontier capability and the rest do not.

Measure the distribution before splitting anything

The premise of a multi-model setup is that your requests differ in difficulty. Verify it before building routing, because occasionally they do not.

Take a sample of production requests, run them through a cheap model, and score the outcomes with the same binary acceptance test you use elsewhere. The result is a success rate for the cheap model on real traffic, and the shape of the failures tells you where the boundary is.

Usually the pattern is legible. Short, well-specified, single-step tasks succeed. Long-context, multi-step, ambiguous ones fail. That legibility is what makes routing possible; if failures are scattered randomly across request types, routing has nothing to key on and you should stay with one model.

Record cost alongside outcome, because the whole exercise is about cost per successful outcome rather than per-token price. Cost-adjusted scoring covers the metric.

Split by task, not by request

The most reliable routing key is not a classifier looking at the request. It is the code path the request came from.

You already know that this call is generating a commit message and that call is planning a refactor. Those are different endpoints in your own application, and hard-coding a different model per endpoint is trivially simple, perfectly predictable, and needs no extra inference.

Typical split: commit messages, changelog entries, simple classification and log triage go to a cheap fast model. Code review comments and test generation go to a mid-tier model. Multi-step agent work and large-repository reasoning go to the strongest model you can afford. Commit message generation and choosing for code review cover the ends of that range.

This captures most of the available saving with almost none of the complexity, and it is where every multi-model setup should start.

Escalation as the second pattern

When a single endpoint has mixed difficulty, the alternative is to try cheap first and escalate on failure.

The cheap model attempts the task. A validator checks the result — JSON parses, test compiles, patch applies, tool arguments match the schema — and on failure the request is retried against the stronger model. Users see one interaction; you pay the high price only on the fraction that needs it.

The arithmetic works when two conditions hold. The cheap model must succeed on a clear majority, and the validation must be cheap and reliable. If the cheap model succeeds forty percent of the time you are paying for two calls on most requests and the saving evaporates.

Latency is the cost you pay in exchange. An escalated request takes both calls sequentially, so the tail of the latency distribution gets worse even though the median improves. For interactive work, check the p95 before shipping it. Latency-adjusted scoring covers setting that budget.

Classifier routing, and why to avoid it early

The sophisticated version routes with a small model that predicts difficulty from the request itself. It is appealing and it is the pattern most likely to be regretted.

The classifier is another model call, with its own latency, cost and failure modes. It needs its own evaluation set and its own monitoring, and when it drifts, every downstream metric moves for reasons that are hard to attribute. Model drift in production covers detecting that.

It also has a systematic weakness: predicting difficulty before attempting a task is genuinely hard. Escalation gets to observe an actual failure, which is far more information than any prefix-based prediction.

Use a classifier only when task-based routing is impossible and escalation latency is unacceptable. That combination is rarer than it sounds.

What multi-model setups cost you

Every model in the system is a prompt to maintain, an evaluation set to keep current, a deprecation calendar to track and a set of failure modes to learn. Three models is three times that burden, not one and a bit.

Prompts do not transfer cleanly between models, so a change to shared instructions has to be validated everywhere. Teams frequently discover this after the third time a fix landed for one path and not the others.

Observability gets harder too. Every trace, log line and cost record needs the served model identifier attached, or aggregate metrics become uninterpretable the moment routing proportions shift. Agent observability and tracing covers the instrumentation.

There is also a floor below which the saving does not justify any of this. If your monthly spend is small, the engineering time to build and maintain routing costs more than the tokens it saves. Budgeting for small teams puts numbers around that.

A staged plan

Start with one model and instrument cost and outcome per endpoint. You cannot route sensibly without knowing where the spend is, and the answer is often concentrated in one or two paths.

Then move the highest-volume, lowest-difficulty endpoint to a cheaper model and measure the success rate. If it holds, take the next one. This captures the large majority of realistic saving with a few configuration values.

Add escalation only where a single endpoint has genuinely mixed difficulty and you have a cheap validator. Add a classifier only if you have exhausted both and can show the numbers justify it. Model routing and fallbacks covers the plumbing that makes all three the same mechanism.