The Electricity Cost of Inference: Estimate It Yourself

Ask what a thousand tokens costs in electricity and the answers you find span orders of magnitude. Nobody is lying. The number depends on four inputs, three of which are properties of your deployment rather than of the model.

So the useful output here is not a figure. It is an equation you can fill in with your own hardware, your own utilisation and your own tariff, plus a clear sense of which term actually moves the answer.

The equation

Energy per token is average device power draw divided by tokens per second of throughput. Multiply by your electricity price and you have a cost per token; scale to a million and you have something comparable to an API price sheet.

Written out: cost per 1M tokens = (watts / tokens_per_second) * (1e6 / 3.6e6) * price_per_kWh. The middle term converts watt-seconds into kilowatt-hours. Nothing else is required for a first estimate.

Every quantity in it is one you must supply. Board power comes from the vendor specification for the accelerator you actually rent or own. Throughput comes from a benchmark you run yourself, at your batch size and your sequence lengths. The tariff comes from your utility bill or your cloud region's published rate.

Resist the temptation to borrow a throughput number from a vendor blog post. It will have been measured at whichever batch size and sequence length flattered the hardware, and throughput is the term the result is most sensitive to.

A worked example, with every assumption labelled

Assume — and replace all three with your own measurements — an accelerator drawing 700 watts under load, sustaining 2,000 output tokens per second across a healthy batch, at an electricity price of $0.15 per kWh. These are placeholders chosen to make the arithmetic legible, not measurements of any real system.

700 watts over 2,000 tokens per second is 0.35 watt-seconds per token. A million tokens is 350,000 watt-seconds, which is 0.097 kWh. At the assumed tariff that comes to roughly 1.5 cents per million output tokens.

Now change a single input. Drop throughput to 200 tokens per second, which is plausible for a large model served at batch size one, and the same million tokens costs about 15 cents. Same hardware, same tariff, ten times the energy — purely because the accelerator spent most of its time waiting.

That sensitivity is the entire lesson. Energy per token is not a property of a model. It is a property of how busy you manage to keep the silicon.

The terms people leave out

Accelerator draw is not facility draw. The host node, networking, storage and cooling all consume power, and data centres express the ratio as PUE. Your provider or facilities team publishes theirs; multiply your accelerator figure by it rather than guessing.

Idle time is the larger omission. A GPU held ready for traffic that has not arrived still draws meaningful power, and if your service is provisioned for a peak it sees twice a day, most of the day is idle draw amortised over very few tokens.

This is the same failure that wrecks self-hosting business cases in dollar terms, and it fails identically in joules. The break-even calculation for self-hosting is worth working through alongside this one, because both hinge on the same utilisation figure.

Why the number is far below your API bill

Even the pessimistic branch above lands at cents per million tokens, against published API rates that run from around $0.14 per million input tokens at the budget end to $3 for a frontier model. Electricity is not what sets those prices.

What sets them is capital cost of accelerators amortised across their useful life, the utilisation risk the provider absorbs on your behalf, engineering, networking, support and margin. Energy is a real line item and a small one.

That matters because energy arguments are often deployed as if they explain pricing. They do not. If you want to understand why one model costs twenty times another, look at active parameter count, memory footprint and how many concurrent requests a serving stack can pack onto a device.

Architecture changes the arithmetic

Mixture-of-experts models activate a fraction of their parameters per token. Kimi K3 is 2.8T total against 104B active, DeepSeek V4 Pro is 1.6T against 49B, GLM-5.2 is roughly 744B against 40B. Compute and memory traffic per token track the active figure, not the total.

The total still bites, because the whole weight set has to be resident somewhere. Kimi K3's open weights are around 1.6TB, which sets how many accelerators must be powered and cooled even when nothing is being served.

Dense models invert the trade. Qwen 3.6 27B is a dense 27B that fits on a single GPU, so its idle footprint is one device rather than a rack. The dense versus MoE comparison covers the serving consequences, and quantisation is the other lever on resident memory.

Prefill and decode are not equally expensive

Reading a prompt is compute-bound and highly parallel. Generating a response is memory-bandwidth-bound and strictly serial, one token at a time. Per token, output is substantially more energy-hungry than input on the same hardware.

That is the physical reason price sheets charge more for output than input across every provider — a five-fold gap on Kimi K3, at $3 in against $15 out. The billing structure reflects the machine, not a marketing decision.

It also means a very long prompt is cheaper per token than the short answer it produces, which is counterintuitive until you have watched the utilisation graphs. Inference latency covers the same split from the timing side.

Measuring your own figure

Sample device power at a fixed interval through a representative production window using your vendor's telemetry, integrate to get joules, and count tokens served over the identical window. Divide. That is your real number, PUE aside.

Do it under production traffic rather than a synthetic loop. A benchmark loop keeps the device saturated in a way real request arrival never does, and the gap between the two is exactly the idle overhead you are trying to quantify.

The decision rule: if the figure you get is within a factor of two of your API spend, you have a genuine hosting case worth pursuing. If it is two orders of magnitude below, energy was never the constraint, and your effort belongs in reducing token usage instead.