The Cost of AI in CI: Why Pipelines Multiply Your Spend

An AI step in CI is the easiest spend to underestimate, because the thing you approve is one model call and the thing you pay for is that call multiplied by your commit frequency, your branch count and your build matrix.

Nobody decides to run a review model two hundred times a day. It happens because a workflow triggers on push, twelve developers push, and each push runs the job on four platform legs. The unit cost was fine; the multiplier was never discussed.

The multiplier, worked through

Start with one pipeline run. A review step that reads a diff, some surrounding context and a project conventions file might send 15,000 input tokens and produce 1,500 output tokens. That is a small number, and it is the number people budget from.

Now apply the multipliers. Ten engineers pushing an average of six times a day is sixty runs. Add a nightly job on three long-lived branches. Add re-runs after flaky failures, perhaps fifteen percent. You are at roughly seventy-five runs a day, or around 1,600 a month.

That turns 15,000 tokens into about 24 million input tokens a month from a single workflow step, before any matrix expansion. The step did not get more expensive; you simply ran it 1,600 times.

Do this arithmetic before merging the workflow. Multiply per-run tokens by pushes per developer, by developers, by branches, by matrix legs, by a re-run factor. The result is usually large enough to change the trigger you were about to write.

Matrix builds are the sharpest edge

A build matrix exists to test the same code against several environments. That is exactly what a model call does not need.

If your job runs across three operating systems and three runtime versions, a naively placed AI step runs nine times per push and produces nine near-identical outputs, nine near-identical comments and nine identical charges. The information gained on legs two through nine is zero.

The fix is mechanical: hoist the model step into its own job that does not inherit the matrix, and have it depend on the matrix jobs only if it needs their results. This one change routinely cuts a workflow bill by most of its size, and it costs a few lines of YAML.

The same logic applies to any per-shard test job. Model calls belong once per commit, not once per shard.

Why it stays invisible

CI spend hides better than interactive spend for three reasons, and all three are worth closing.

It runs unattended, so nobody is watching the latency or the token counter the way they would in an editor. It is charged to a shared service key rather than a person, so per-user dashboards show nothing. And its growth tracks team activity rather than a decision — a hiring round or a busy release week raises the bill with no change to any configuration.

Give CI its own API key, so the spend is attributable at the source rather than reconstructed later. Then alert on the daily total for that key rather than on the monthly account figure, because a runaway workflow reaches a meaningful number in hours. Alerting on token spikes covers thresholds that catch it in time.

Which checks are worth a model call

The test is simple: could a deterministic tool answer this? If yes, use the tool. It is free, instant, repeatable and does not need review.

Formatting, linting, type errors, dependency vulnerabilities, licence checks, dead code, import cycles and test failures all have mature deterministic tooling. Asking a model to look for them is paying for a worse version of something you already have installed.

What is left is genuinely judgement-shaped work: does this change match the intent described in the pull request body, is there a subtle logic inversion the type system cannot see, does this migration have a rollback path, is this error message going to help the person who reads it at three in the morning. Cost per code review covers sizing that job properly.

Summarisation is the other honest category — release notes, changelog drafts, a plain-language description of a large diff. These produce output a human consumes directly, and no deterministic tool substitutes.

Gating strategies that actually bound it

Ordered by how much they save relative to effort.

Trigger on pull request, not on push. This is the single largest cut available. A developer pushes many times per pull request and opens one. Running on pull request open and on subsequent review-ready updates, rather than on every commit, typically removes three quarters of the runs.

Skip drafts. Work in progress does not need review comments. Gate on the pull request being marked ready.

Filter by path and size. No model call for changes confined to documentation, lock files or generated code. Skip diffs below a few lines, where the signal is not worth the call, and skip enormous ones where the output will be too shallow to act on — flag those for human attention instead.

Cache by content hash. Key the result on a hash of the exact diff plus the prompt version. Re-runs after infrastructure flakes then cost nothing, and re-runs are a large fraction of total runs on most teams. Prompt caching covers the provider-side version, which stacks with this.

Cap concurrency and cancel superseded runs. If three pushes land in five minutes, only the last one needs the model. Most CI systems support cancelling in-progress runs for the same ref in one setting.

Choose the model for the job, not the marquee

CI checks are mostly classification and summarisation over a bounded input, which is where smaller and cheaper models perform closest to frontier ones.

The reasonable approach is to run your candidate models over fifty real diffs from your own history, compare the outputs against what human reviewers actually flagged, and pick the cheapest one that holds up. Reach for a stronger model only on the narrow checks where the cheap one demonstrably misses things. When a cheap model is enough covers making that call without guessing.

Also pin the model version explicitly. An unpinned alias can change under you, altering both output quality and per-run token counts without any commit on your side.

Make it structurally incapable of blocking

This is a reliability point with a cost consequence. If the model step can fail the build, an outage or a rate limit turns into an engineering-wide stoppage, and the response is usually to retry the pipeline — which spends more money on top of the disruption.

Advisory steps should carry a timeout, a failure mode that skips silently, and no ability to set a required status. Running LLMs in CI covers the security and reliability side of the same workflow, including why fork pull requests need separate handling.

The checklist before you merge the workflow

Estimate runs per month by multiplying pushes, developers, branches, matrix legs and a re-run factor, then multiply by per-run tokens. Move the step out of the matrix. Trigger on pull request rather than push, skip drafts, and filter by path. Cache on the diff hash. Give CI its own key and alert on its daily total.

Then track cost per merged pull request rather than cost per pipeline run, because runs on abandoned branches are pure loss and only the merged-PR figure tells you what the automation is worth. Cost per pull request covers building that number.