The Real Cost of Migrating Between Model Providers

A provider migration gets scoped as a configuration change because the configuration change is the visible part. Base URL, API key, model string. An afternoon, maybe less if you already route through a gateway.

Then the estimate is wrong by a factor of ten, because the actual work is everything downstream of the call: prompts that were tuned against the old model, evaluations that only have baselines for it, tool schemas shaped around its parser, and a period where you are paying for both providers at once. This article gives you a structure to estimate that properly before you commit.

Step zero: the evaluation you probably do not have

You cannot measure a migration you cannot evaluate, and most teams discover at this point that their evidence the system works is a mix of production metrics and impressions.

Before anything else, you need a fixed task set with either expected outputs or graded criteria, and a harness that runs it against an arbitrary model. If that exists, this step is a day. If it does not, budget one to three weeks, because assembling a representative task set is the slow part, not the code.

This is not migration overhead you can skip. Without it you will cut over on vibes, and the regression will surface in production a fortnight later when nobody is looking for it. Building a model eval harness covers the shape; benchmarking on your own work covers choosing the tasks.

Prompt re-tuning: count prompts, not systems

Every production prompt that has been through more than a couple of iterations carries corrections specific to the old model. Removing them and discovering what the new model needs instead is the single largest line item in most migrations.

Estimate per prompt, not per service. A simple extraction prompt with a tight schema might be half a day including evaluation. A long system prompt driving an agent, with accumulated behavioural instructions, is commonly three to five days before its scores match the old baseline, and sometimes they never quite do on every axis.

Multiply by your actual prompt count. Teams routinely find they have forty prompts when they thought they had six, because every feature quietly added one. Grep for the string that starts your system messages before you estimate anything.

One shortcut that genuinely works: migrate in order of traffic share. The top three prompts usually account for most of the spend and most of the risk, and completing those gives you a real measurement of your per-prompt rate to extrapolate from.

Tool schemas and agent loops

If your workload calls tools, this is the second largest item and the one most likely to be underestimated, because the schemas look declarative and portable.

The things that differ in practice: strictness of JSON Schema adherence, whether the model emits several tool calls at once, behaviour when a required argument is genuinely unavailable, and how it responds to a tool returning an error rather than a result. Each of those is a place where your agent code has an assumption baked in.

Budget time for schema changes and for the loop code around them separately. The schemas are usually a day or two; the loop — retry conditions, argument validation, the phrasing of correction messages — is where the tail sits. Tool schema design and tool call retries cover the failure modes worth testing explicitly.

Streaming, errors and the unglamorous layer

Providers differ in how they stream: chunk granularity, where usage totals appear in the stream if they appear at all, how tool calls are represented mid-stream, and what happens on a mid-response disconnect.

They also differ in error semantics. Status codes for rate limiting, whether a retry-after header is present and honoured, how context-length overruns are reported, and which failures are transient versus permanent. Your backoff logic encodes assumptions about all of it.

None of this is hard work. It is a week of small surprises, and it almost always lands on someone who was not scoped into the migration. Handling SSE disconnects and the error code reference cover what to check.

Add tokenizer differences here too. The same text does not produce the same token count across models, so your context budgets, truncation points and cost forecasts all shift slightly. Tokenizer differences across models covers the size of the effect.

The dual-run period is a real budget line

Any migration you can defend involves running both providers concurrently for a while — shadow traffic first, then a percentage split, then cutover.

During shadow running you pay for every request twice. If shadowing runs for two weeks at full traffic, that is two weeks of duplicated spend, plus the engineering time to build the comparison tooling if you do not have it.

The percentage-split phase is cheaper in tokens but longer in calendar time, because you are waiting for enough production volume to detect a difference. Two to four weeks is typical before anyone is comfortable going to a hundred percent.

You can cut this cost by shadowing a sample rather than all traffic — ten percent is usually enough to catch structural differences — and by shadowing only the highest-value paths. What you cannot safely cut is the period itself.

A worked estimate structure

Fill this in with your own numbers. Everything is in person-days except the two spend lines.

Engineering days. Eval harness build or validation, E days. Prompt re-tuning, P prompts times D days each. Tool schema and agent loop work, T days. Streaming, error handling and tokenizer adjustments, S days. Comparison tooling for the dual run, C days. Cutover, monitoring and rollback readiness, R days. Total engineering days = E + (P x D) + T + S + C + R.

Duplicated spend. Shadow phase: your daily token spend, times the fraction of traffic shadowed, times the number of shadow days. Split phase: the difference between old and new rates on the split portion, which may be negative if the new provider is cheaper.

Risk allowance. Add a contingency for the case where the new model fails on a workload the old one handled — typically the cost of keeping the old provider live for one more month plus the days to route that workload back.

Then compare the total against the annualised saving. A migration that saves thirty percent on a modest monthly bill and costs sixty engineering days does not pay back inside a year, and that is a perfectly good reason not to do it — or a reason to do it only for the one workload where the saving is concentrated.

What makes migrations cheap

The teams that move quickly are not the ones with clever abstractions. They are the ones who kept four things in order beforehand.

A model-independent evaluation suite on their own tasks. Prompts in versioned files with model-specific workarounds labelled as such. Tool schemas written to the specification rather than to one model's parser. And a gateway where the provider is a configuration value, so the split phase is a routing rule rather than a deployment. Model routing and fallbacks covers that last piece.

If those are in place, a migration is genuinely a couple of weeks. If none of them are, the first migration is expensive and the second is cheap, because doing it once forces you to build all four. That is worth knowing when you scope the first one: a meaningful fraction of the cost is one-time investment in portability, not migration work as such. Measuring lock-in costs covers estimating that exposure before you are forced to move.

The decision rule

Do not migrate on rate card alone. Compute cost per successful task on both providers using your own evaluation, since a cheaper per-token model that needs more retries or emits longer outputs can lose on total.

Migrate when the annualised saving exceeds the total estimate above by a comfortable margin, when the new provider clears your evaluation on the workloads that matter, and when you have somewhere to fall back to. Otherwise, keep the option and revisit when either the price gap or your volume grows.