Inference vs Training Cost: Which One Is Actually Your Problem

Training and inference are both called compute and behave nothing alike. One is a single enormous batch job that happens before you ever see the model; the other is a stream of small latency-sensitive requests that continues for as long as anyone uses it.

Almost every confused conversation about AI economics comes from mixing them up. As a developer, you pay for exactly one of them.

What each one is

Training runs once. A cluster processes an enormous corpus repeatedly, computing gradients and updating weights, for weeks or months. It is throughput-bound, entirely batch, tolerant of latency, and it can be checkpointed and resumed. The output is a fixed set of weights.

Inference runs every time someone sends a request. It reads a prompt, produces tokens, and finishes. It is latency-sensitive, unpredictable in arrival, and it must be served continuously with capacity provisioned for peaks it cannot control.

The costs also fall differently in time. Training is capital: spent up front, amortised over every future request. Inference is operational: proportional to usage, never amortised, and it grows exactly as fast as your product succeeds.

Why the arithmetic differs so sharply

A single inference request is trivially cheap next to a training run. The aggregate is not, because the multiplier is unbounded.

Training touches every parameter and computes gradients, which is several times the work of a forward pass alone, and it repeats over an entire corpus. Inference does one forward pass per token — and for mixture-of-experts models, only the active parameters participate, so a model with trillions of total weights may compute against a small fraction of them per token. Mixture of experts explained covers that routing.

The consequence is that a successful model spends more compute serving requests over its lifetime than it did being trained. This is not a marginal comparison; it is why the industry's hardware demand is dominated by serving rather than by training runs.

It is also why inference efficiency gets so much engineering attention. A percentage point off training cost saves money once. A percentage point off inference cost saves money on every request forever.

Which one you actually pay

If you call an API, you pay only inference. Training cost is embedded in the price the way a factory is embedded in the price of a component, and you cannot influence it, negotiate it or optimise it.

What you can influence is your own token consumption, which is the entire surface of your bill. Prompt length, output length, retry behaviour, how much of a conversation you resend, whether reasoning is enabled, and whether your prefixes cache.

Cached input is the sharpest lever, because a cache hit skips prefill for the shared prefix entirely. Kimi K3 prices cached input at $0.30 per million against $3 uncached — a tenfold difference for the same tokens, decided purely by how you structure the prompt. Prompt caching explained covers getting those hits reliably.

If you self-host open weights, you still pay only inference. You have not trained anything; you are running someone else's finished weights on hardware you rent or own, which converts a per-token price into a per-hour one. Self-hosting versus managed API cost covers where that break-even sits.

Fine-tuning is the only training you might buy

Fine-tuning is the one place a normal team touches training cost, and it is far smaller than pretraining — a modest amount of data and a short run rather than a corpus and a cluster.

It is still usually the wrong first move. Prompting and retrieval solve most problems that people initially reach for fine-tuning to solve, they iterate in minutes rather than hours, and they do not leave you owning an artefact that has to be retrained every time the base model moves.

Fine-tuning earns its cost when you need a consistent output format at very high volume, when a small model must acquire a narrow skill it lacks, or when shrinking the prompt saves more per request than the training run costs. Fine-tuning versus prompting versus RAG covers the decision honestly.

The cost that catches teams out is not the run itself but the recurring one. A fine-tuned model is frozen against a base that keeps improving, and serving it often costs more per token than serving the shared base model.

Why the distinction shapes the prices you see

Inference economics explain most of the pricing structure you deal with.

Input and output are priced differently because prefill and decode have different characteristics — prefill is parallel and compute-bound, decode is sequential and memory-bandwidth-bound. Output typically costs several times input as a result. Input versus output token pricing covers the ratio.

Cached input is cheap because the provider genuinely skips work. Long context costs more because the KV cache occupies memory for the whole session, limiting how many sessions a machine can hold. KV cache explained covers that occupancy.

And open-weight releases make sense economically precisely because training is a sunk cost. Publishing weights does not increase what the training run cost, while it can build an ecosystem around the lab's hosted inference. DeepSeek V4 and GLM-5.2 shipping under MIT, and Kimi K3 releasing weights under Moonshot's own terms with a revenue threshold for model-as-a-service, are variations on that calculation.

What to do with this

Budget for inference and ignore training entirely unless you are seriously considering a fine-tune. Your controllable cost is tokens, and it is fully visible in your own logs.

Instrument per-request token usage before optimising anything, because intuition about where tokens go is wrong more often than not — retries, resent conversation history and reasoning tokens routinely dominate the parts people focus on. Token accounting for finance covers turning that into a number a finance team accepts.

Then optimise in the order that pays: cache stable prefixes, cap output length, stop resending history you have already summarised, and route easy work to a cheaper model. Reducing token usage covers the rest.