Pinning Model Versions: Why Aliases Break Production Quietly

Most providers offer two ways to name a model: a stable alias that tracks the current version, and a dated identifier that points at fixed weights. Defaults usually favour the alias, and that default is wrong for anything running in production.

The failure it produces is unusually unpleasant because nothing errors. Output quality shifts, and every symptom points somewhere other than the cause.

What actually happens

You build against an alias. You tune the prompt over several weeks until the output is reliable — the phrasing that stops it being verbose, the example that fixes the edge case, the instruction that keeps the JSON shape stable.

The provider updates what the alias resolves to. The new version is better on aggregate benchmarks, and it responds differently to your prompt. Formatting drifts. An instruction that used to be obeyed is now interpreted more loosely. Your parse rate falls two percent.

Nothing throws. Nothing appears in an error log. The regression surfaces as a slow rise in support tickets or a quiet increase in retries, days or weeks after the change, and the first three hypotheses anyone tests are all wrong.

Why tuned prompts are fragile

Prompt tuning is fitting to a specific model's behaviour. That is not a criticism — it is what tuning is — but it means the result is a function of the model, not a property of the prompt.

The tighter the fit, the more fragile it is. A prompt that says "return JSON" transfers well. A prompt with six carefully ordered instructions, two negative examples and a specific phrasing that suppresses a particular failure mode is tuned to one model's quirks, and those quirks are exactly what a new version changes.

Teams with the most carefully optimised prompts are therefore the most exposed. That is worth knowing, because it inverts the usual intuition that more engineering means more robustness.

Pin the version, schedule the upgrade

The fix is straightforward. Use dated or explicitly versioned model identifiers in production, so the weights behind your application only change when you change them.

Then treat model upgrades as deliberate work. When a new version ships, run your evaluation set against it, compare results, adjust prompts if needed, and deploy the change as a change — with a rollback available.

This converts an invisible, unattributable regression into a scheduled task with a known blast radius. That is the entire value of pinning, and it is large.

The cost is that you must actually do the upgrades. A pinned version eventually gets deprecated, and a team that pinned two years ago and never revisited has a migration rather than an upgrade. Model deprecation and migration covers staying ahead of that.

Pin more than the model name

The model is the biggest source of drift, not the only one.

Sampling parameters matter — temperature, top-p, and any penalties. Provider defaults can change, and a default shift produces the same silent quality drift as a model change. Set them explicitly rather than relying on defaults, even where the default is what you want. Temperature, top-p and sampling covers what each does.

Prompts themselves should be versioned alongside code, with the model version they were tuned against recorded. When someone asks why output changed, being able to diff both is what makes the question answerable.

System-level context — injected dates, retrieved documents, tool definitions — changes behaviour too, and is frequently modified by people who do not realise they are changing model output.

Detecting drift when you cannot pin

Some providers do not offer pinned versions, and some proxies resolve aliases outside your control. In that case, detection is the fallback.

Keep a small set of canary prompts with known-good outputs and run them on a schedule — a handful of representative requests, checked for structural and semantic properties rather than exact string equality. When a canary result shifts, you have a timestamp and a diff, which turns an unattributable regression into a bounded investigation.

Track structural metrics in production as well: parse success rate, average output length, refusal rate. These move before anyone files a ticket, and a step change in any of them is a strong signal that something upstream changed. Model drift in production covers what to watch.