Cost-Adjusted Model Scoring: Ranking by Value, Not Capability

Every public leaderboard ranks capability and stops there. That is a reasonable scope for a benchmark and a terrible basis for a purchasing decision, because the models at the top can cost an order of magnitude more than the models three rows down.

Cost-adjusted scoring is the obvious correction, and it is easy to do badly. The naive version — divide the benchmark score by the price — produces a ranking that is arithmetically valid and practically useless.

Why score divided by price fails

Benchmark scores are not ratio quantities. A model scoring 57 on a composite index is not "thirty percent better" than one scoring 44; the scale is normalised and the zero point is arbitrary. Dividing by price treats it as though a score of zero means no capability, which it does not.

The deeper problem is that capability is not linear in value. On many tasks there is a threshold below which output is unusable and above which extra capability adds little. A model that solves eighty percent of your tasks and one that solves eighty-two are close to interchangeable; a model that solves forty is worthless regardless of price.

Ratios also hide magnitude. A very cheap, very weak model wins any score-per-dollar contest and would be a poor choice for almost any real workload. The metric has to be bounded by an absolute capability floor or it recommends nonsense.

Score the task, not the model

The construction that works starts from your workload rather than the leaderboard. Take a representative set of your own requests, run each candidate, and record two numbers per request: whether the outcome was acceptable, and what it cost.

Acceptable is a binary you define. Did the test pass, did the patch apply, did the JSON parse, did the reviewer accept it. Binary outcomes are far more robust than graded scores and much easier to agree on. Benchmarking on your own work covers assembling the set.

Then the metric is cost per successful outcome: total spend divided by the number of acceptable results. This is a real ratio quantity with a meaningful zero, it is denominated in currency, and it automatically penalises a cheap model that needs three attempts to get one usable answer.

Retries are where cheap models lose

The cost-per-success framing surfaces something that per-token price lists cannot. A model at a fifth of the price that fails twice as often is not five times cheaper; after retries, the gap narrows sharply and sometimes inverts.

Retries also cost more than one call, because a failed attempt usually consumes the full input context before producing a bad output, and the retry pays for that context again. The hidden cost of retries works the arithmetic through.

In agent loops the effect compounds. A weaker model takes more steps to reach the same end state, and every step re-sends an ever-growing conversation. The per-token saving is consumed by step count long before the task finishes. Why agent costs are unpredictable covers that dynamic.

Input, output and cache are different prices

A single blended price per million tokens is the wrong unit for comparison, because the ratio between input and output pricing varies substantially and your workload has its own ratio.

Kimi K3 lists $3 in and $15 out per million, a five-to-one spread, with cached input at $0.30. DeepSeek V4 Pro lists roughly $0.44 in and $0.87 out — a two-to-one spread. A workload that reads a large repository and emits a short answer is dominated by input pricing; an agent that generates long patches is dominated by output.

Compute your actual token ratio from logs before comparing anything. It is a single query and it frequently reverses a preference formed from headline prices. Input versus output token pricing covers the asymmetry in detail.

Cached input deserves separate treatment. If your prompts share a long stable prefix, the effective input price is much lower than the list price, and models with aggressive cache pricing move up the ranking. Prompt caching savings math shows how to estimate the hit rate.

The floor and the ceiling

Two constraints turn a ratio into a decision rule.

The floor is a minimum acceptable success rate. Below it, no price makes the model viable, and it is excluded regardless of what the ratio says. Set this from what your users or your pipeline can tolerate, not from a benchmark.

The ceiling is a budget per unit of work — per pull request, per developer per month, per task. Above it, a model is excluded even if it is the best available. Selecting for cost ceilings covers working backwards from a fixed budget.

Between floor and ceiling, rank by cost per successful outcome. Outside them, the model is simply not a candidate. This is a much more honest structure than a single leaderboard, and it produces different answers for different teams, which is correct.

What the score still does not capture

Latency is absent, and for interactive work it can dominate. A model that is cheaper per success and three times slower may be the wrong choice in an editor and the right one in a nightly batch. Latency-adjusted scoring handles that axis separately.

Variance is absent too. Two models with the same cost per success can have very different tail behaviour, and a model that occasionally burns a hundred thousand tokens on a runaway loop is a budgeting problem the average conceals.

Switching cost is absent. A modest improvement in cost per success does not pay for prompt re-tuning, re-evaluation and new failure modes. Fold that in before acting on a small margin, and treat anything under about twenty percent as a tie.