GPU Rental Cost Math: Utilisation Decides Everything

GPU rental converts a variable cost into a fixed one. That single change is what makes self-hosting attractive at scale and punishing below it, and most of the confusion around the decision comes from comparing a fixed cost to a variable one without accounting for utilisation.

The unit that matters

An API charges per token. A rented accelerator charges per hour. To compare them at all you need effective cost per token, which requires knowing how many tokens you actually push through the hardware.

Effective cost per token is the hourly rate divided by tokens produced in that hour. The hourly rate is fixed and knowable. The tokens produced are entirely determined by your traffic, and that is the term people guess at.

The consequence is stark. At full utilisation, effective cost per token can be very low. At ten percent utilisation it is ten times higher for identical hardware. The hardware did not change; the denominator did.

Traffic shape is the hidden variable

Almost no real workload is flat. Developer tooling follows working hours, concentrated in a handful of hours across five days. Batch jobs spike and go quiet. Customer-facing traffic has daily and weekly cycles.

A workload concentrated in eight hours a day, five days a week, uses roughly 24 percent of a week's hours. If capacity is sized for peak, three quarters of what you rent is idle — and you pay for all of it.

This is the structural advantage of per-token pricing that fixed-cost comparisons ignore. An API charges nothing at 3am on Sunday. That is not a minor convenience; for a spiky workload it is the whole argument.

Before computing anything, plot your request volume by hour across a representative fortnight. The shape of that graph decides the answer more than any per-token rate.

Peak or average

Sizing for peak means paying for peak capacity all the time. Sizing for average means queueing during peaks, which shows up as latency.

Neither is wrong. Which is right depends on whether anyone is waiting. Batch work can queue happily and should be sized near average. Interactive work cannot, and must be sized nearer peak — which is exactly the workload where idle cost is highest, because interactive traffic is the spikiest.

The uncomfortable conclusion is that the workloads best suited to self-hosting on cost grounds are the ones that can tolerate queueing, and those are frequently the same workloads that qualify for discounted batch API pricing. Batch API savings math covers that alternative.

Throughput is not what the benchmark said

Published tokens-per-second figures are usually measured with a single request and no competition. Real throughput at concurrency is lower, sometimes substantially.

Three things eat into it. KV cache memory is consumed per concurrent session and grows with context length, so long-context work sharply limits how many sessions fit. Batching helps throughput but adds latency. And mixture-of-experts models have less predictable throughput than dense ones, because uneven routing across experts leaves some overloaded and others idle.

Measure at your real concurrency and your real context lengths. A deployment sized from single-request benchmarks will disappoint under load, and the gap is not small. KV cache explained covers the memory side.

Costs beyond the hourly rate

Redundancy, because one accelerator is a single point of failure and doubling it doubles the hardware line. Storage and egress, which are minor but not zero. And engineering time to build, maintain, upgrade and be paged for the deployment, which frequently exceeds the hardware cost outright and is the term most often left out.

Include all three or the comparison is not honest. The self-hosting break-even point works through the full calculation.

The calculation to run

Take your measured hourly request volume across two weeks. Multiply by average tokens per request to get tokens per hour, by hour. Decide whether you are sizing for peak or average and pick a capacity. Compute what that capacity costs for a full month at your provider's rate, add redundancy and a realistic engineering allocation, and divide by total monthly tokens.

Compare the result to your API rate. If it is not clearly better by a comfortable margin, stay on the API — the calculation has enough uncertainty in it that a narrow win on paper is not a win in practice.