Model Drift in Production: Detecting Quality Change With No Error

Most production failures announce themselves. A service returns 500s, a queue backs up, a dashboard turns red. Model quality regression does none of that. The API returns 200, the response is well-formed, and it is worse than it was last week.

This is the failure mode that makes language model operations different from ordinary service operations, and almost nobody instruments for it until they have been burned once.

Where drift comes from

There are four sources, and they need separating because the fixes differ.

The model changed. You are calling an alias and the provider updated what it resolves to, or a pinned version was silently routed to a successor at deprecation. This is the classic case and the reason to pin model versions in the first place.

Your input changed. The model is identical; the distribution of what you send it moved. A new customer segment with different document formats, a codebase that grew past the point where your context assembly truncates usefully, a retrieval index that quietly stopped updating.

Your context changed. Somebody edited a system prompt, added a tool definition, changed the injected date format, or altered how documents are concatenated. These edits are often made by people who do not think of themselves as changing model behaviour. System prompts are load-bearing infrastructure and rarely treated as such.

Sampling changed. Provider defaults for temperature or top-p can move, and a default shift produces exactly the same symptom as a model change. Set them explicitly.

The signals that move first

You do not need semantic evaluation to detect most drift. Structural metrics move earlier and are cheap to compute on every request.

Track parse success rate if you consume structured output. A model that starts wrapping JSON in prose, or emitting a trailing explanation, will show up here within hours rather than the weeks it takes for a support ticket to arrive.

Track output length distribution, not the mean. A model that becomes more verbose shifts the whole distribution right, and the tail moves before the average does. Percentiles catch this; averages hide it.

Track tool-call validity — the fraction of tool invocations whose arguments match the schema — and refusal rate, the fraction of responses that decline the task. Both are single-digit percentages in healthy operation and both step changes are unmistakable when plotted. Agent observability and tracing covers where to put the hooks.

Canary prompts

Structural metrics tell you something changed. Canaries tell you what.

Keep a small fixed set of representative requests — twenty to fifty is plenty — with known-good outputs recorded. Run them on a schedule against production configuration, not a test harness, so they exercise the same prompts, tools and routing as real traffic.

Check them for properties rather than exact equality. Exact string matching fails constantly for reasons unrelated to quality, and a check that cries wolf gets muted. Assert the shape: valid JSON, required fields present, the right tool selected, length within a band, a specific fact appearing in the answer.

The value of canaries is a timestamp. When a metric drifts, the canary history tells you the day it started, which bounds the set of changes worth investigating from months to hours. That is the difference between a bounded investigation and a hypothesis-generation exercise.

Separating your change from their change

When something moves, the first question is whose fault it is, and the answer is usually available if you kept the right records.

Version prompts alongside code, and record which model version each was tuned against. A deploy log and a metric step change with the same timestamp is a strong answer. A metric step change with no deploy near it points upstream.

Log the model identifier the provider actually served, not the one you requested, if the API returns it. Alias resolution and silent successor routing both become visible immediately when you record what came back rather than what you sent.

Keep a fraction of traffic on a second model as a reference line. If both models degrade together, the cause is your input or your context. If only one moves, it is the model. Model routing and fallbacks gives you the machinery for this at close to zero extra effort.

Semantic checks, used sparingly

Eventually you want to know whether answers are still correct, not merely well-shaped. Automated semantic evaluation is the tool, and it deserves scepticism.

Using a model to grade another model introduces its own drift, since the grader is also a model that can change. Pin the grader, and re-validate it against human-labelled examples periodically, or the monitor becomes the thing you cannot trust.

A cheaper approach is to instrument the outcomes you already have. Did the generated test pass. Did the SQL execute. Did the patch apply. Did the user accept the suggestion or edit it heavily. These are free, unambiguous, and much better correlated with real quality than any grader.

A minimum viable setup

If you instrument nothing else, instrument four things: parse success rate, output length percentiles, tool-call validity and the served model identifier. All four are computed from data you already have and cost effectively nothing.

Add twenty canary prompts on a daily schedule, checked for structural properties, with results retained for at least a quarter so you have a baseline to compare against.

Then set one alert: a step change in parse success rate. It is the single metric most likely to catch a real regression first, and a team that has only that is dramatically better off than one relying on users to notice.