Attention Mechanisms Explained Without the Linear Algebra

Attention is the mechanism that made transformers work, and it is usually explained with matrices that obscure what it is doing. The idea underneath is simple, and understanding it explains most of the practical limits you run into — why long context is expensive, why prompts degrade in the middle, and why caching helps so much.

The problem it solves

To predict the next token, a model needs to know which earlier tokens matter. In the sentence "the server returned a 502 because the upstream had crashed", predicting what follows "crashed" depends heavily on "server" and "upstream" and barely at all on "the".

Earlier architectures processed sequences in order, carrying a running summary forward. That summary was a bottleneck: everything had to be compressed into fixed-size state, and information from far back got diluted or lost.

Attention removes the bottleneck by letting every position look directly at every other position. Nothing has to survive a long chain of summarisation.

What it computes

For each token, the model produces three vectors: a query, a key and a value. The query represents what this token is looking for. The key represents what each token offers. The value is the content it contributes if selected.

The model compares each query against every key, producing a relevance score for every pair. Those scores are normalised into weights that sum to one, and the output for a position is a weighted blend of all the values.

That is the whole mechanism. Each token asks a question, every token advertises what it has, and the answer is a weighted mixture of what matched.

Multi-head attention runs several of these in parallel with different learned projections, so one head can track syntactic agreement while another tracks which variable a name refers to. The heads are concatenated and mixed.

Why it costs what it costs

Every token compares against every other token. For a sequence of length n, that is n squared comparisons.

Double the input and attention work quadruples. This is the single most important practical consequence of the architecture, and it explains a family of behaviours that otherwise look arbitrary.

It is why long prompts are disproportionately slow to process rather than merely proportionally slow. It is why time to first token climbs sharply with prompt length. It is why providers price input tokens at all, when intuitively reading ought to be cheap. And it is why an enormous amount of engineering effort has gone into making attention cheaper without changing what it computes.

LLM inference latency explained covers how this shows up in wall-clock time.

The optimisations you benefit from

Several techniques attack the cost without changing the mathematics, which is why they are invisible in output quality.

KV caching stores the keys and values already computed for previous tokens. During generation, each new token only computes its own key and value and reuses the rest, which turns an O(n squared) regeneration into an incremental step. This is the reason generation is much faster than initial prompt processing. KV cache explained covers the memory cost, which is substantial.

Flash attention reorganises the computation to avoid writing the full score matrix to slow memory, processing it in blocks that fit in fast on-chip memory. Same result, considerably less memory traffic.

Grouped-query attention shares keys and values across multiple query heads, cutting the memory the cache consumes. This is one of the main reasons long context windows became affordable.

Why the middle of a long prompt is weaker

Attention weights are normalised to sum to one across all positions. With ten tokens competing, meaningful weight can go to several. With five hundred thousand competing, the weight available for any individual token is tiny unless it stands out sharply.

Training compounds this. Models see far more short sequences than very long ones, so the ability to pick out a specific token at extreme distance is less well practised than nearby attention.

The result is the well-documented pattern where information at the start and end of a long prompt is used more reliably than information in the middle. It is not a bug in any particular model; it follows from how the mechanism and its training interact. The lost-in-the-middle problem covers working around it.

What this means practically

Three things follow directly and are worth acting on.

Put important instructions at the beginning or the end of a long prompt, never buried in the middle. This costs nothing and measurably improves adherence.

Prefer retrieving twenty thousand relevant tokens over supplying eight hundred thousand mostly-irrelevant ones. It is cheaper, faster, and produces better attention on what matters.

Structure prompts so the stable prefix comes first and variable content last, which maximises what prompt caching can reuse. Prompt caching explained covers the specifics.