Cost Per 1K Lines of Code: A Tempting, Mostly Bad Metric

Somebody in every organisation eventually divides the monthly model bill by the number of lines the tooling produced. The number that falls out is concrete, comparable across months, and easy to put on a slide. It is also close to meaningless for almost every decision you would want to make with it.

The problem is not that the arithmetic is wrong. It is that lines of code sits at the wrong end of the causal chain: it measures the volume of the artefact rather than the work that produced it or the value it delivers.

What the metric actually measures

Cost per thousand lines is total spend divided by net lines added. Both halves of that fraction are unstable in ways that have nothing to do with efficiency.

The numerator is dominated by input tokens, not output. An agent that reads forty files, runs the test suite three times and then writes twenty lines has spent nearly all of its budget on reading and reasoning. The twenty lines are the residue of the work, not the work.

The denominator is a formatting artefact. The same logic written with one statement per line, trailing commas and a brace on its own line can be double the line count of a compact version. Run a formatter with different settings and your cost per thousand lines halves without a single token changing hands.

It rewards verbosity, immediately and predictably

Any metric that appears on a dashboard eventually becomes a target, and this one has a trivial exploit: produce more lines.

Models are already biased towards verbose output. Ask for a function and you often get defensive null checks, a docstring restating the signature, and three commented alternatives. Under a cost-per-line metric all of that is free improvement to the score, and the reviewer who deletes it is making the number worse.

The same inversion hits engineers. A developer who spends an afternoon with a model working out that the right change is deleting a class and calling an existing helper has produced negative lines at positive cost, which the metric records as infinitely bad. That is precisely the work you want more of.

Lines and tokens are not proportional

Even if you accepted lines as a unit of output, the conversion to tokens is not constant across a codebase.

Tokenisers split on subword boundaries, so a line of dense Python with common identifiers costs far fewer tokens than a line of Java with long camel-case names, or a line of minified configuration, or anything containing non-ASCII text. How tokenisation works explains the mechanism, and token costs by language covers how wide the spread gets in practice.

The consequence is that comparing cost per thousand lines between two services in different languages tells you about their tokenisers, not their teams. Comparing across months tells you about which part of the codebase happened to get touched.

Where it genuinely is useful

There is a narrow band where the metric works, and it is worth naming because the blanket dismissal is also wrong.

The condition is mechanical, homogeneous, high-volume output: a framework migration across four hundred similar files, a codemod that a model applies with judgement at each site, generated API clients from a schema, or a bulk translation of test fixtures. In those jobs the lines really are the unit of work, they are uniform in shape, and there is no verbosity incentive because the target form is fixed.

Here the metric earns its keep as a forecasting tool. Run the job on thirty files, measure spend per file, multiply. The estimate holds because the population is homogeneous, which is the assumption that fails everywhere else. Choosing a model for migrations covers the model side of the same problem.

The other legitimate use is a cheap sanity check on a batch job. If a bulk transformation suddenly costs three times as much per file as last week, something changed — a longer prompt, a retry loop, a larger context. It is a smoke alarm, not a performance review.

The deletion problem

Net lines can be zero or negative for the most valuable work, and there is no repair for this within the metric.

Some teams switch to churn — added plus deleted — to dodge the sign problem. That is worse, because it now rewards rewriting a file rather than editing it, and models are already inclined to reproduce a whole file when a three-line patch would do. You have moved the exploit rather than closed it.

Any denominator built from diff size shares this defect. The fix is to stop counting the diff.

Better denominators

Pick a unit that corresponds to finished work rather than emitted text.

Cost per merged pull request is the strongest general-purpose figure. It counts only work that survived review, it is comparable across languages, and it is naturally resistant to verbosity because a bloated diff makes review slower rather than the number better. Cost per pull request covers how to attribute spend to one.

Cost per completed task works where tickets are reasonably uniform — bug fixes, support escalations, small features. It captures the reading and debugging that produced no lines at all.

Cost per successful run is the right unit for automation, because it charges failed attempts to the same denominator instead of hiding them. The hidden cost of failed runs works through that accounting.

Whichever you choose, track the distribution rather than the mean. AI spend is heavily skewed, and the tail is where the fixable problem lives.

The decision rule

Use cost per thousand lines only when the work is bulk, mechanical and uniform, and only as a forecast or an alarm — never as a measure of how well a team or a model is doing.

For everything else, divide by merged pull requests or completed tasks, and pair it with a quality signal such as review rework rate. A cost figure without a quality figure beside it can always be improved by shipping worse work faster. Measuring AI ROI for developers covers building that pair.