Alerting on Token Spikes Before the Invoice Explains Them

LLM spend has an unpleasant property: it can rise by an order of magnitude in an afternoon and stay invisible until the invoice arrives weeks later. A deployment that accidentally doubles agent concurrency, a retry loop with no ceiling, a prompt change that quadruples input — none of these throw errors.

Rate, not total

The most common mistake is alerting on cumulative monthly spend against a budget. By the time that fires, the money is gone.

Alert on rate instead: tokens per hour, or spend per hour, compared against the same window in recent history. A tenfold rise in hourly consumption is actionable within the hour, while a monthly total crossing 80 percent tells you only that the month is nearly over.

Keep the cumulative alert as a backstop. It catches slow drift that never spikes, which rate alerts miss by construction.

What to measure

Four signals cover most failure modes, and they fail differently enough that you want all four.

Tokens per hour, by workload. Aggregate spend hides which system moved. Tagging requests by service or job type turns "spend tripled" into "the nightly review job tripled", which is a fixable statement.

Tokens per unit of work. Tokens per agent run, per pull request, per processed document. This separates "we are doing more work" from "each piece of work got more expensive". The first is usually fine; the second is usually a bug.

Cached input fraction. A sudden drop means something broke your prompt prefix — a variable moved above the stable content, or a prompt version diverged between code paths. The bill can multiply severalfold with no other visible change. Prompt caching savings math covers why.

Failed and retried request share. A rising retry rate consumes budget while producing nothing, and it usually indicates an upstream problem worth knowing about anyway. The hidden cost of retries covers the accounting.

Thresholds that survive contact with reality

Static thresholds fail in both directions. Set high enough not to fire on Monday morning traffic, they miss real problems on a quiet Sunday.

Compare against the same hour of the same weekday over the past few weeks. LLM workloads are strongly periodic — tied to working hours, CI schedules and deploy cadence — so a day-of-week-aware baseline removes most false positives without losing sensitivity.

Alert on a multiple of the baseline rather than an absolute number, so the alert keeps working as the workload grows. Something in the range of three to five times the baseline for the hour tends to catch genuine runaways without firing on ordinary variation, but derive yours from your own history rather than taking a number from an article.

Hard limits, not just alerts

An alert requires someone to be awake. A limit does not.

Set a per-session token ceiling so no single agent run can consume unbounded budget. Set a fleet-level rolling cap so a deployment change that triples concurrency queues work rather than spending three times as much. And where the provider offers a hard spend cap, use it as a final backstop.

The distinction between these matters: a session ceiling protects you from one pathological task, a fleet cap protects you from a configuration change, and a provider cap protects you from everything you did not think of. Agent token budgets covers setting the first two from data.

Keeping alerts credible

An alert that fires weekly and is ignored weekly is worse than no alert, because it trains people to dismiss the category.

Route by severity. A rate spike with a hard cap already engaged is informational and belongs in a channel, not a page. A rate spike with no cap and rising is a page. Include enough context in the alert to act — which workload, what the baseline was, what changed — so the first ten minutes are not spent working out where to look.

And review fired alerts monthly. If a threshold has fired six times and been dismissed six times, it is wrong and should be changed rather than tolerated.

Attribute spend before you need to

The hardest part of responding to a spike is working out what caused it, and that work is nearly impossible to do retrospectively if requests were not tagged when they were made.

Attach metadata to every request at the point of origin: the service, the job type, the environment, and where practical a run identifier. Providers that return usage in the response make it straightforward to accumulate these locally even when their own dashboards do not support the breakdown.

With attribution in place, a spike alert can name the responsible workload directly rather than sending someone to grep deploy logs for anything that changed. Without it, every investigation starts from zero and takes hours rather than minutes.

This is worth doing before you have a problem, because it cannot be backfilled. The tokens that were spent yesterday are already unattributed and will stay that way.

Test the alerting path

An untested alert is an assumption. Deliberately generate a spike in a staging environment and confirm the alert fires, reaches a human, and carries enough information to act on.

The common failures are mundane: the metric was never actually being exported, the threshold was set against a unit different from the one being emitted, or the notification routed to a channel nobody watches. All three look identical to a working system right up until the moment they matter.