Long Context Degradation: How Quality Falls Before the Limit

The intuitive model of a context window is a container. Below the limit everything works; above it the request is rejected. Nothing in between.

The observed behaviour is a slope. Accuracy on a fixed task declines as the prompt grows, gradually and well before any limit is reached. The model does not announce this. It answers with the same confidence at 400K tokens as at 4K, and the answers are worse.

Why attention dilutes

Each generated token attends over every earlier token, distributing a finite amount of weight across them. As the number of candidates grows, the share available to any one relevant token shrinks unless the model can sharpen the distribution enough to compensate.

Sharpening is exactly what gets harder in a long, homogeneous prompt. Ten thousand tokens of similar-looking code all present a moderate match to a query about a function name. The correct one has to win against a large field of near-misses rather than a small one.

This is why degradation depends so much on the content, not only on the length. A long prompt made of clearly distinct sections degrades less than a long prompt of uniform material at the same token count. Attention mechanisms explained covers the underlying operation.

Position is part of the picture

Degradation is not uniform across the prompt. Content near the beginning and near the end is used more reliably than content in the middle, and the middle region widens as the prompt grows.

The practical effect is counterintuitive: adding more context can make the model worse at using context you already gave it, because material that was near the end has been pushed into the middle. Teams often discover this by appending one more document and watching an established behaviour break.

Put the material that matters most last. It costs nothing and it reliably helps. The lost-in-the-middle problem covers the effect and the ordering strategies.

Instruction adherence decays before retrieval does

The most common production symptom is not a missed fact. It is the model quietly abandoning the format or constraint it was given.

A system prompt that reliably produces JSON at 5K tokens starts emitting a prose preamble at 200K. A style rule holds for the first several turns of a long conversation and then relaxes. The instruction is still in the window, still technically attended to, and increasingly outcompeted by the volume of material that followed it.

This decays earlier than retrieval accuracy because instructions are short and the competing content is long. Restating critical constraints near the end of the prompt, immediately before generation, is the cheapest mitigation available. System prompts explained covers placement.

Where agents feel it first

An agent loop accumulates context by construction. Every tool call and every result is appended, so a long-running task grows its own prompt until quality drops on its own.

The visible pattern is a competent agent that becomes repetitive after some number of steps — re-reading files it already read, re-attempting fixes it already tried, losing track of the original objective. That is degradation, not a reasoning failure, and adding more capable models does not remove it.

The fix is context management rather than a bigger window: summarise completed phases, drop stale tool output, and keep the working set small. Context compaction strategies covers the techniques, and agent memory and context management covers the architecture around them.

Measuring the slope for your own task

You cannot manage this without a curve, and the curve is cheap to produce. Fix a question set with known answers. Run it at several context sizes — 8K, 32K, 128K, 300K — padding with real but irrelevant material from your own corpus.

Plot accuracy against size. You get a slope, not a cliff, and the point where it crosses your acceptable threshold is the number your pipeline should be designed around. That number is almost always well below the advertised window.

Run the curve separately for each task shape you care about. Simple lookup, multi-fact synthesis, and format adherence decline at different rates, and a mixed pipeline is governed by the steepest one. Needle-in-a-haystack tests explains why the standard public version of this measurement is too easy to substitute for it.

The cost dimension

Degradation is not the only reason to keep prompts short. Input tokens are billed on every request, and a large fixed context resent on each turn multiplies fast.

Latency scales with prompt length too, because the whole input is processed before the first output token appears. A pipeline that felt responsive at 20K can feel slow at 300K even when the answers are still correct. Context window cost trade-offs covers the arithmetic.

The awkward conclusion is that the same lever fixes both problems. Trimming context improves accuracy, cost and latency simultaneously, which is rare enough to be worth acting on deliberately rather than when something breaks.

A working rule

Treat the advertised window as a hard ceiling and your measured curve as the design budget. Fill the prompt with material that is relevant, not material that is available.

When a long-context task starts producing worse answers, check prompt size before checking the model. Look for the point where the prompt grew, and try the same task with the irrelevant portion removed. In most cases the model was fine and the prompt was not. RAG versus long context covers when to stop growing the prompt and start selecting for it.