KV Cache Explained: The Memory Behind Long Context

The KV cache explains two things that otherwise look contradictory: why a model takes several seconds to read your prompt and then produces tokens rapidly, and why a long conversation gets more expensive to continue even when each new message is short.

What it stores

Attention gives each token a key and a value vector. When generating the next token, the model needs the keys and values of every previous token in order to attend over them.

Those vectors do not change. The key and value for token fifty are the same whether the model is generating token fifty-one or token five thousand. Recomputing them on every step would be enormous waste.

So they are computed once and kept. That store is the KV cache, and it is the difference between generation being practical and being impossible.

Two phases with different characteristics

Prefill processes your prompt. Every token's key and value is computed, and attention runs across the whole input. This is where the quadratic cost of attention lands, and it is why time to first token scales sharply with prompt length. Prefill is compute-bound and parallelisable — all the input tokens are known, so they can be processed together.

Decode generates output one token at a time. Each step computes one new key and value, appends them to the cache, and attends over everything stored. This is cheap in computation but strictly sequential, because each token depends on the previous one.

Decode is memory-bandwidth-bound rather than compute-bound. The bottleneck is moving the cache and the weights, not doing arithmetic on them. This is why generation speed depends more on memory bandwidth than on raw compute, and why the two phases benefit from different optimisations.

The memory cost is the real constraint

Cache size scales with sequence length, number of layers, number of attention heads and head dimension. Every additional token adds a fixed amount for every layer.

At long context this becomes the binding constraint on serving. A model whose weights fit comfortably in memory can still fail to serve many concurrent long-context requests, because each session's cache is held separately for its whole lifetime.

That is the operational reality behind long context being expensive. It is not primarily that the computation is hard — it is that memory is occupied per session, which limits how many sessions a machine can hold at once. Providers pricing long context higher are pricing that occupancy.

Why architectures changed to shrink it

Several now-standard design choices exist mainly to reduce cache size.

Grouped-query attention shares one set of keys and values across several query heads instead of giving each head its own. This cuts cache size by the sharing factor with modest quality cost, and it is a large part of why 1M-token context became affordable.

Multi-head latent attention, used in several 2026 models including the Kimi line, compresses the cached representation further. Kimi K2.6 uses MLA specifically to make its context practical at its parameter count.

When a model announces a very long context window, some cache-compression technique is nearly always what made it possible.

Prompt caching is this, persisted

Provider-side prompt caching applies the same idea across requests rather than within one.

If two requests share a prefix — the same system prompt, the same tool definitions, the same document — the KV cache for that prefix can be computed once and reused. The second request skips prefill for the shared portion entirely.

The saving is large for chat and agent workloads, where a long stable prefix is followed by a short changing suffix. This is why providers price cached input dramatically below uncached input, and why prompt structure affects your bill.

The practical rule follows directly: put stable content first and variable content last. A prompt that injects the current timestamp at the top invalidates the entire cache on every request. Moving that to the bottom preserves it. Prompt caching explained covers the details.

What to do with this

Order prompts stable-first, variable-last, deliberately. Treat it as a design constraint rather than an afterthought, because it is one of the few optimisations that improves latency and cost simultaneously.

Expect long conversations to get slower and more expensive per turn even when each message is short, because the cache grows monotonically. Compaction is the fix — summarise and drop, rather than accumulating indefinitely. Agent memory and context management covers it.

If you self-host, size hardware for concurrent cache occupancy at your real context lengths, not for the weights alone. This is the calculation most commonly got wrong, and it produces a deployment that benchmarks well with one request and falls over under load.