The Lost-in-the-Middle Problem: Position Beats Relevance

Place a fact near the beginning of a long prompt and the model will use it. Place the same fact in the middle and it may not. The content is identical; only the position changed.

This is one of the most practically important behaviours of long-context models, and it is routinely designed around badly — usually by assuming that anything inside the context window is equally available.

The shape of the effect

Retrieval accuracy plotted against position in the prompt is not flat. It is high at the start, drops through the middle, and rises again toward the end, producing a shallow U.

The effect strengthens as prompts get longer. At a few thousand tokens it is barely measurable. At several hundred thousand it is pronounced enough to change whether a pipeline works.

Crucially it applies to models with very large advertised windows just as much as to smaller ones. A million-token window means the model will accept a million tokens, not that it attends evenly across them.

Why it happens

Two mechanisms combine.

Attention weights are normalised to sum to one across all positions. With a handful of tokens competing, several can receive meaningful weight. With hundreds of thousands competing, any individual token receives very little unless it stands out strongly. Dilution is structural. Attention mechanisms explained covers the mechanism.

Training distribution compounds it. Models see vastly more short sequences than extremely long ones, and within long documents, beginnings and endings carry disproportionate significance — titles, abstracts, summaries, conclusions. The model learns that the edges matter, because in its training data they usually did.

Neither is a defect anyone chose. Both follow from how the architecture and the data interact, which is why the pattern appears across models from different labs.

Measuring it on your own data

Do not rely on published figures. The effect varies by model, by prompt length and by how distinctive your content is, so measure it where it matters.

Take a document representative of your real inputs. Insert an unambiguous fact the model cannot infer from surrounding context — a specific identifier, an unusual value — at 10, 25, 50, 75 and 90 percent depth. Ask for it back. Repeat at several total lengths: 50K, 200K, 500K.

The resulting grid tells you where your model stops being reliable on your data. That number is worth more than any vendor's context claim, and it takes an afternoon to produce.

Two refinements make it sharper. Test reasoning over two distant facts, not just retrieval of one, since combining is harder than finding. And use content that resembles yours — code behaves differently from prose, and a benchmark built on prose will mislead you if you work with code.

Arranging prompts around it

Instructions at the edges. Put system instructions at the top and repeat the critical ones at the bottom, immediately before the model generates. The final instruction before generation is the most reliably followed position in the whole prompt.

Most relevant material last. If you are supplying retrieved documents, order them with the highest-scoring chunk nearest the end rather than the beginning. This inverts what many retrieval pipelines do by default.

Never bury a constraint. A safety rule or a formatting requirement placed in the middle of a long prompt is the least likely instruction to be followed. If it matters, it goes at an edge.

The better fix is usually less context

Prompt arrangement mitigates the problem. Reducing prompt length removes it.

A model given the twenty thousand relevant tokens will outperform the same model given eight hundred thousand tokens containing the same twenty thousand. It is cheaper, faster, and the attention is concentrated where it should be.

This is the strongest practical argument for retrieval over stuffing, and it holds even when the whole corpus would technically fit. The window being large enough is not a reason to fill it. RAG vs long context covers when retrieval is genuinely worse, which is a narrower set of cases than its advocates on either side suggest.

For agent loops the equivalent is compaction: summarise completed work and drop the raw tool output rather than letting the transcript grow monotonically. The transcript is the prompt, and an unmanaged one drifts into exactly the regime where the middle stops being read. Agent memory and context management covers the mechanics.