GPU Memory for Inference: What Actually Fills the Card

The usual way to size a self-hosted deployment is to work out how much memory the weights need, find hardware with at least that much, and stop. That calculation produces a server that works beautifully for one request and collapses under ten.

Weights are the fixed part of the budget. The variable part is what decides how many users you can serve, and it is the part people leave out.

The weights, which are the easy part

Weight memory is parameters multiplied by bytes per parameter. At 16-bit precision that is two bytes each; at 8-bit, one; at 4-bit, roughly half. The arithmetic is simple and it is the one number everybody computes correctly.

For a mixture-of-experts model, the number that matters here is total parameters, not active. Every expert has to be resident even though only a few participate in any given token, because routing decisions are made per token and there is no way to know in advance which experts a request will need.

This is why MoE models are cheap to run and expensive to host. Kimi K3's open weights are around 1.6TB, which settles the self-hosting question for most teams immediately — the compute per token is modest, and the memory to hold the model is not. Dense versus MoE models compared covers that asymmetry.

Quantisation is the standard lever here, cutting weight memory at some cost in quality that varies by model and by how aggressively you push it. Quantization explained covers where the trade-off turns bad.

The KV cache, which is the hard part

Every token in an active session has keys and values stored for every layer, and they stay resident for the whole session. That store grows with sequence length, with the number of layers, and with the number of concurrent sessions.

The critical property is that it scales with concurrency multiplied by context length. Ten users at short context is a different memory profile from ten users at long context, and both are different from a hundred users at either.

At the context lengths current models advertise, this dominates. Kimi K3, GLM-5.2, both DeepSeek V4 variants and MiniMax M3 all ship 1M-token windows. A handful of sessions genuinely using those windows can consume more memory than the weights do, on a model whose weights already filled the card. KV cache explained covers the mechanism.

This is the line item that turns a working demo into a failed deployment, and it is the reason architectural work like grouped-query attention and multi-head latent attention exists at all — both are cache-shrinking techniques, and Kimi K2.6 uses MLA specifically to make its 256K window practical.

The lines people forget

Three further consumers sit between the weights and the cache, and together they are not small.

Activations, the intermediate tensors produced during a forward pass, scale with batch size and sequence length. They are transient rather than persistent, but they must fit at peak, and a large batch of long prompts is exactly when peak arrives.

Framework and runtime overhead — the CUDA context, memory allocator fragmentation, communication buffers when a model is sharded across devices. This is unglamorous and routinely accounts for a real slice of the card.

Headroom. A deployment running at ninety-nine percent memory does not degrade gracefully; it fails the request that crosses the line. Serving frameworks handle this by capping concurrency and queueing, which converts an out-of-memory error into latency, but only if you have configured the cap correctly.

Why concurrency runs out before compute does

The mismatch that surprises people is that a GPU is usually far from compute-saturated when it stops accepting work.

Decode is memory-bandwidth-bound, not compute-bound: each generated token requires reading weights and cache from memory and doing relatively little arithmetic. The arithmetic units idle while memory traffic saturates. LLM inference latency explained covers that split.

Meanwhile every additional concurrent session claims cache memory permanently for its lifetime. So the ceiling you hit is a memory ceiling, expressed as a maximum number of concurrent sessions at your typical context length, while utilisation graphs still show the compute mostly idle.

Batching is what recovers that idle compute, since processing several sequences together amortises the weight reads across them. But batching needs memory for activations and for every batched sequence's cache, so it trades the resource you are short of for the one you have spare. Batch size and throughput covers where that trade stops paying.

How to size it honestly

Work from your traffic, not from the model card.

Start with your real numbers: peak concurrent requests, median and worst-case context length, and required latency. Those three decide everything downstream, and guessing any of them makes the rest of the exercise decorative.

Compute weight memory at the precision you will actually deploy. Then estimate cache memory at peak concurrency and realistic context length — not the advertised maximum, which almost nobody uses, but the length your logs show. Add activations at your intended batch size, add runtime overhead, and keep meaningful headroom.

Then test it. Load-test at peak concurrency with realistic prompt lengths, not with short synthetic ones, because short prompts hide the cache problem completely. A benchmark run with hundred-token prompts tells you nothing about a deployment serving fifty-thousand-token ones.

What to do when it does not fit

Four options, roughly in order of how much they cost you.

Quantise the weights. This is usually the first move and often the only one needed, though it should be validated on your own tasks rather than on published perplexity numbers.

Cap context. Enforcing a shorter maximum than the model supports is unglamorous and extremely effective, because cache memory scales directly with it and most requests do not need the full window. RAG versus long context covers retrieving less instead of sending more.

Shard across devices, accepting the communication overhead and the added operational complexity.

Or use a smaller model. Qwen 3.6 27B is dense, runs on a single GPU and reaches 77.2% on SWE-bench Verified, which is enough for a great deal of production work at a memory profile that fits ordinary hardware. Best model for self-hosting covers choosing among the realistic candidates.