Budgeting for Experimentation Without a Surprise Bill

Production AI spend is boring and predictable: known workloads, known volumes, a forecast that lands within twenty percent. Experimental spend is neither, and it is where nearly every shocking invoice comes from.

The reason is not that experiments are expensive. Most cost very little. The reason is that experiments have no natural ceiling — a production workload is bounded by real traffic, while a curious engineer with a loop and a large corpus is bounded by nothing at all.

Why unbudgeted experimentation produces the worst bills

Three patterns account for most of the damage, and all three are structural rather than careless.

The first is the evaluation sweep. Someone wants to compare four models across two hundred test cases with three prompt variants. That is 2,400 calls, which sounds modest until each one carries a large context. The comparison is worth doing; the arithmetic simply was never done first.

The second is the left-running job. A batch script written to process a sample gets pointed at the full dataset, or a nightly cron is created for a prototype that nobody remembers to delete. It keeps billing after the person who wrote it has moved on to something else.

The third is the runaway loop. An agent prototype without a turn limit, retrying a failing tool call, or two agents delegating to each other. This one can generate a month of normal spend in an afternoon, and unlike the other two it produces nothing at all. Agent costs are unpredictable for exactly this reason: nothing in the loop tells it to stop.

Separate the ledgers before anything else

The single most useful control is not a limit, it is a boundary. Experimental spend and production spend must be distinguishable at the source, which means separate API keys at minimum and preferably separate billing projects.

Mixed ledgers cause two specific failures. A production forecast built from mixed data is wrong, because it includes one-off spikes that will not recur. And a spike investigation starts with an hour of archaeology to work out whether the anomaly was a customer surge or somebody trying something on a Tuesday afternoon.

Once separated, the two get different treatment. Production spend gets forecasting, capacity planning and optimisation work. Experimental spend gets a cap and a review — you do not optimise it, because the point of an experiment is to find out something, and micro-optimising the cost of learning is a poor use of anyone.

AI spend forecasting covers the production half; this article is about the other one.

Time-box rather than token-box

The instinct is to give an experiment a token budget. It is the wrong unit for the same reason story points beat hours for estimating: nobody knows the number in advance, so the limit is arbitrary.

Token caps also fail in a specific and annoying way. They terminate work mid-sweep, so you have paid for sixty percent of an evaluation and learned nothing, because a partial comparison across four models is not a comparison. The money is spent and the finding is not obtained, which is the worst of both outcomes.

A time-box behaves better. "Two days on this, then write up what you found" gives a stopping point that arrives at a natural boundary, forces the result to be recorded, and bounds the spend well enough in practice because a person can only launch so many jobs in two days. Pair it with an absolute ceiling on the key as a backstop against the runaway loop, set high enough that it never fires during normal work.

The one place a token cap is right is inside the experiment itself: a per-run turn limit on an agent prototype, which stops a single loop rather than the project. Agent token budgets covers setting those.

What a useful experiment budget looks like

Concretely: a fixed share of total AI spend, ring-fenced, with no approval required below a per-person threshold.

Ten to fifteen percent of the total is a defensible starting point for a team actively adopting these tools; five percent for one in steady state. The exact figure matters less than it being a real number that exists before anyone asks.

The no-approval threshold is the part that gets skipped and the part that does most of the work. If a developer must ask permission to spend the price of a lunch on finding out whether a cheaper model handles their workload, they will not ask — they will keep paying the expensive model indefinitely, which costs far more than the experiment would have. Approval friction on small amounts reliably costs more than it saves.

Above the threshold, the ask should be one paragraph: what question this answers, roughly how many calls, and what decision changes based on the result. That last clause filters out most of the genuinely wasteful proposals on its own.

The shadow spend problem

Where experimentation is hard to fund, it does not stop. It moves onto personal cards, free tiers and unmanaged accounts.

That is worse on every axis. The spend becomes invisible, so your forecast is wrong. Company code goes through accounts with no data agreement. The results stay with the individual instead of the team. And people quietly stop doing the evaluation work that keeps model choice current, so you carry an outdated default for another year.

A funded, low-friction experiment budget is partly a cost control and partly a governance control, and the governance half is usually the more valuable one.

Kill switches worth having

Three mechanisms, all cheap to set up, that close the failure modes above.

Expiring keys. Issue experiment keys with an end date matching the time-box. This kills the left-running job automatically, which is the failure mode that persists longest because nobody notices it.

A hard provider-side cap. Not an alert — an actual stop. Alerts are for gradual drift; a runaway loop outruns any human response time. Alerting on token spikes covers what to catch with each.

A monthly orphan sweep. List every key with activity, match against a named owner and a live purpose, revoke the rest. This takes fifteen minutes and consistently finds something.

Flat-rate access changes this calculus, since experiments do not add to a per-token bill — that is the honest case for it, and it is also why ProjectCOZY exists. It shifts the constraint from money to time, which is easier to manage but does not remove the need for the orphan sweep. Per-token versus flat-rate pricing compares the models.

Require a write-up, not a justification

The output of an experiment budget is knowledge, and knowledge that stays in one person is knowledge you will buy again in six months when somebody else asks the same question.

Ask for a short, permanent record: the question, the method, the numbers, the conclusion, and the date. Five paragraphs is plenty. The date matters more than it looks, because model capability moves fast enough that a finding from last quarter deserves re-testing rather than reuse. Evaluating models on your own codebase covers making those results reproducible.

Negative results count and should be recorded with the same weight. "We tried routing this workload to a smaller model and quality dropped on these three case types" is a finding that saves the next person two days.

The setup, in one pass

Ring-fence a percentage of AI spend for experiments. Issue separate, expiring keys so the ledger is clean and the jobs die on schedule. Set a per-person no-approval threshold. Time-box the work rather than capping tokens, with an absolute provider-side stop as the backstop. Sweep for orphaned keys monthly. Require a dated write-up in a shared place.

That is a morning of setup, and it converts the most volatile line on your AI bill into a fixed, planned one. Building an AI budget for a small team covers where this fits in the wider picture.