AI Cost Anomaly Detection That Catches Real Problems

An LLM cost anomaly is not a gradual thing. Spend rises by a factor of five in twenty minutes because a loop lost its exit condition, or a prompt change moved a variable above the cached prefix, or someone shipped a fan-out that spawns forty subagents per request instead of four. None of these throw an error. The pipeline reports success the entire time.

Anomaly detection here is a narrow problem with a narrow solution. You are not trying to find every unusual pattern in your billing data. You are trying to catch, within an hour, the small set of failures that turn a normal week into a five-figure surprise, and to do it without generating an alert every Monday at nine.

What a spike looks like in the underlying data

Spend is a derived number. The primitives are input tokens, output tokens, cached input tokens and request count, and each anomaly has a distinct fingerprint across those four.

A runaway agent loop shows a rising request count with a roughly flat cost per request, then rising input tokens per request as the transcript accumulates. Output stays modest because each turn produces a short tool call. A caching regression shows the opposite: request count is unchanged, output is unchanged, and uncached input jumps by whatever fraction of your prompt used to be a stable prefix.

A context-bloat regression shows flat request counts with input per request stepping up once and staying there — someone started attaching a file, or a retrieval step widened its window. A reasoning-model switch shows output tokens per request rising several-fold with input unchanged. Because the shapes differ, an alert that fires on the four primitives separately tells you what happened; one that fires on total dollars only tells you that something did.

Baseline per workload, not per account

Account-level spend is the sum of things with completely different behaviour: a nightly batch job, an interactive assistant, a CI review bot, and a handful of developers experimenting. Their variances add. The combined series is noisy enough that any threshold sensitive to a single workload doubling will fire constantly on ordinary movement in the others.

The fix is to baseline each tagged workload separately. A nightly job with near-constant volume can carry a tight band — a 50 percent deviation is genuinely abnormal. An interactive workload driven by human activity needs a much wider one. Applying one global sensitivity to both means either missing the batch job's failures or drowning in false positives from the interactive one.

This requires that requests carry a workload tag before they reach the provider, which is the same prerequisite as everything else useful in this area. Agent cost attribution covers how to attach and propagate it.

Why absolute thresholds fail

The instinct is to set a ceiling: alert if today exceeds some number of dollars. This fails in both directions and gets worse over time.

Set the ceiling high enough that a busy Tuesday does not trip it, and a quiet Sunday can absorb a tenfold anomaly without reaching it. Set it low enough to catch the Sunday case and it fires every Tuesday until someone silences it. Worse, a static ceiling has to be revised every time the team grows, and nobody remembers to revise it, so it either becomes permanently noisy or permanently inert.

Rate-of-change thresholds do not have this problem because they are scale-free. Comparing the current hour against the median of the same hour on the same weekday for the past four weeks gives you a ratio, and a ratio of three means the same thing on a quiet Sunday as on a busy Tuesday. Use the median rather than the mean, or a single previous incident will inflate your baseline and blind you to the next one.

Separating a runaway loop from legitimate growth

Both look like more spend. The distinguishing signal is what happened to cost per unit of work.

If tokens per hour tripled and cost per agent run held steady, you are doing three times as much work. That is a capacity question and possibly good news. If tokens per hour tripled while runs per hour stayed flat, each run got three times more expensive, and that is almost always a defect. This is why unit metrics belong in the alerting path rather than only in the monthly report.

Loops have a further tell: the distribution of turns per run develops a tail that runs into your ceiling. A healthy workload has most runs finishing in a handful of turns with a thin tail. A looping one piles up at whatever maximum you configured, because the loop only stops when something stops it. Alerting on the share of runs that hit the turn ceiling catches this before the token numbers move much. Detecting agent loops covers the loop-side detection in detail.

What to page on, and what to leave to a digest

The test is whether a human woken at three in the morning can do something in the next ten minutes that saves money. If not, it is not a page.

Page on: hourly spend rate above a hard multiple of baseline for a single workload, the share of runs hitting the turn ceiling crossing a few percent, and a collapse in cached input fraction on a workload that normally caches heavily. All three have an immediate action — disable the workload, roll back the deploy, revert the prompt.

Digest daily: week-over-week drift in cost per run, changes in model mix, retry rate creeping up, the ten most expensive runs of the day with links to their traces. These matter and none of them is urgent. Putting them in a page devalues the pages that are real. Alerting on token spikes goes deeper on threshold design.

The anomalies that never spike

Rate-of-change detection is blind by construction to anything that grows smoothly. A prompt that gets ten percent longer each sprint, a retrieval step that returns steadily more chunks, a team that gradually adopts a more expensive model — none of these ever produce a ratio that trips a threshold, and together they can double a bill over a quarter.

Catch these with a slow comparison rather than a fast one: this month's cost per unit of work against the same figure three months ago, reviewed by a person. That is a review item, not an alarm, and it belongs in the same monthly cycle as your forecast. The hidden cost of context bloat covers the most common of these drifts.

A minimal setup that works

You need less machinery than vendors suggest. Log input, output, cached input and request count per workload per hour. Compute a rolling median for the same hour-of-week over four weeks. Alert when the current hour exceeds a multiple of that median and the absolute amount is large enough to be worth waking someone for — the second condition suppresses the noise from small workloads where a tenfold ratio is two dollars.

Add one unit metric per workload, alert on that separately, and send everything else to a daily digest. Review the digest weekly for the first month and tune from real false positives rather than imagined ones. That is a day of work and it catches the failures that actually cost money.