Multi-Head Latent Attention Explained

Multi-head latent attention is the second serious answer to the same problem grouped-query attention was invented for: the KV cache is what limits how many long-context requests a machine can serve, and shrinking it is worth real effort. MLA takes a different route, and the difference explains why some 2026 models can hold very long conversations cheaply while others cannot.

The problem both schemes attack

Every token in a session leaves behind a key and a value vector in each layer, and they stay resident until the request ends. The KV cache is not shared between users, so it scales with concurrency as well as with length.

At short context the weights dominate memory and nobody worries. At 200K tokens with a handful of simultaneous users, the cache can exceed the weights, and the server runs out of room while the GPU sits underutilised.

Both GQA and MLA exist to push that ceiling up. They differ in what they choose to give up.

Two different economies

Grouped-query attention reduces how many key-value sets exist. Query heads are grouped and each group shares one set, so a 64-head model might store eight rather than 64. The stored vectors themselves are unchanged; there are simply fewer of them.

MLA leaves the head structure alone and attacks the size of what is stored. Instead of caching full keys and values, it projects them down into a single compact latent vector per token per layer, and caches only that.

When attention runs, the latent vector is projected back up into the keys and values each head needs. Nothing was thrown away at the head level — every head still gets its own key and value — but the thing sitting in memory between steps is far smaller than what those heads consume.

Trading arithmetic for memory

The reconstruction is not free. Every decode step now includes extra projection work that classic attention did not need.

That trade is favourable on current hardware for the same reason flash attention is. Decode is limited by memory bandwidth, not by arithmetic throughput: the chip finishes its multiplications and waits for data. Adding computation to a phase that is idling, in order to move less data, is close to free.

Careful implementations go further and fold some of the up-projection into the surrounding weight matrices, so part of the reconstruction disappears into work the model was doing anyway.

Why it pairs with sparse models

MLA shows up most often in large mixture-of-experts models, and the pairing is not coincidental.

An MoE model has enormous total parameters but activates a small fraction per token, so it is cheap to run relative to its size. Kimi K2.6 is 1T total parameters with 32B active across 384 experts, eight routed plus one shared. Mixture of experts explained covers why that structure works.

That design shifts the bottleneck. Once activated compute is modest, the cache becomes the thing standing between you and serving many users, and cache compression is where the remaining wins are. Kimi K2.6 uses MLA precisely to make its 256K window practical at that scale.

What it means for the models you use

The visible consequence is pricing and behaviour at length, not benchmark scores. A model with an aggressively compressed cache can offer long context at a price that a classically structured model of similar capability cannot match.

It also affects how well a long session holds up under load. Providers throttle or queue when cache memory runs short, so the same model can feel fast on a quiet endpoint and sluggish on a busy one. Architecture sets how much headroom there is before that happens.

If you are comparing long-context options, the useful comparison is not the advertised window but the price and latency at the length you actually use. The 1M-context comparison lays out how the current models differ, and the Kimi K2.6 guide covers that model specifically.

Where the trade does not pay

MLA is more complex to implement than GQA, and complexity has costs. Kernels have to be written and tuned for it, and a runtime without a good MLA path can be slower than the naive version it replaces. This is a real hazard when self-hosting a model whose architecture your inference stack supports only nominally.

The benefit also depends on where you operate. At short context — chat turns of a few thousand tokens, classification, structured extraction — the cache was never the constraint, and the extra projection work is pure overhead. GQA is entirely adequate there and simpler to serve.

Nor does any cache scheme address the softer problem with long prompts, where a model accepts a million tokens but attends less reliably to the middle of them. Compression changes what fits in memory, not what the model reads carefully.

The takeaway

Read attention architecture as a statement about serving economics. Classic multi-head attention means a large cache and expensive long context. GQA means moderate compression at almost no quality cost. MLA means aggressive compression bought with extra compute, and it appears in models designed to run long sessions cheaply at scale.

For API users this is background that explains the price list. If you self-host, check that your runtime has a properly optimised path for the attention scheme your chosen model uses before you commit to hardware — an unoptimised MLA implementation can erase the entire advantage you selected the model for.