Grouped-Query Attention: Why Long Context Got Affordable

Grouped-query attention is a small architectural change with an outsized effect on what it costs to serve a model. It exists for one reason: the KV cache had become the binding constraint on inference, and shrinking it was worth a measurable amount of quality. Nearly every model released in the last few years uses it or something derived from it.

The constraint it was invented for

During generation the model must attend over every previous token, which means it needs each of their key and value vectors. Recomputing them each step would be absurd, so they are stored. The KV cache is that store.

Its size is the product of sequence length, layer count, head count and head dimension, held for the entire lifetime of a request. It is not shared between users, because every conversation has different content.

That last point is what makes it expensive. A server can hold one copy of the weights and serve many requests from it, but it needs a separate cache per session. At long context, cache memory rather than weight memory decides how many concurrent users a machine supports — and concurrency is what determines cost per token.

Where the redundancy was

Classic multi-head attention gives every head its own query, key and value projections. With 64 heads, that is 64 separate sets of keys and values cached per token per layer.

The point of multiple heads is that each learns to look for something different — one tracking which noun a pronoun refers to, another tracking bracket nesting, another tracking a variable's type. That diversity lives mostly in the queries: what each head is asking for.

The keys and values are closer to a shared description of what each token offers. Duplicating that description 64 times turned out to be far more redundancy than the quality gain justified.

The spectrum from MHA to MQA

Multi-query attention took the idea to its extreme: keep all 64 query heads, but give them a single shared set of keys and values. The cache shrinks by a factor of 64, which is dramatic.

It also hurts. With one shared key projection, every head is forced to score relevance using the same notion of what a token offers, and the specialisation that made multiple heads useful is partly collapsed. Models trained this way were measurably weaker and sometimes less stable to train.

Grouped-query attention sits between the two. The query heads are divided into groups, and each group shares one set of keys and values. Eight groups over 64 heads cuts cache size eightfold while leaving eight genuinely distinct key spaces for heads to specialise within.

What the trade actually buys

The empirical finding that made GQA standard is that quality degrades slowly as you reduce key-value heads, right up until the point where it collapses. There is a broad plateau where an eight-fold or sixteen-fold reduction costs very little.

Meanwhile the savings compound with everything else. A smaller cache means more concurrent sessions per machine, which lowers cost per token. It also means less data to read on every decode step, and since decode is bound by memory bandwidth rather than arithmetic, that translates directly into faster generation. Inference latency explained covers why decode is bandwidth-limited.

You get cheaper serving and faster tokens for a small quality cost. That is an unusually good trade, which is why the industry took it almost universally.

How it relates to the newer schemes

GQA is not the end of the line. The Kimi models use multi-head latent attention instead, which compresses the cached representation into a smaller learned form and reconstructs what attention needs on the fly.

The goals are the same and the mechanisms differ. GQA reduces how many key-value sets you store; latent attention reduces how large each stored item is. Multi-head latent attention explained covers the second approach and where it wins.

Both are why the current generation ships 1M-token windows at prices that would have been impossible with classic multi-head attention. Kimi K2.6 pairs MLA with a 256K window; Kimi K3, GLM-5.2, both DeepSeek V4 variants and MiniMax M3 all advertise 1M.

When this appears in your work

If you use hosted APIs, GQA is invisible. It is baked into the architecture and there is nothing to configure. Its effect reaches you as pricing: the cost of long context is lower than it would otherwise be.

If you self-host, it matters directly and it is the number people most often forget. Sizing a deployment by weight memory alone produces a box that runs one request beautifully and falls over at four concurrent users, because nobody budgeted for per-session cache.

The calculation to do is straightforward in shape: take your realistic average context length, multiply by the per-token cache cost implied by the model's layer and key-value head counts, and multiply by the concurrency you need. Compare that to what is left after the weights. If the answer is uncomfortable, a model with more aggressive cache compression will serve you better than a larger GPU. Choosing a model for self-hosting works through the rest of it.

The takeaway

Treat key-value head count as a serving-cost specification rather than a quality one. When two models look similar on benchmarks, the one with fewer key-value heads or a latent cache will usually be cheaper to run at length and quicker to generate under load.

And when a long conversation slows down turn after turn, the cause is cache growth, not the model degrading. The fix is compaction — summarising and dropping history rather than accumulating it forever.