Context Length vs Effective Context: The Number You Can Use

Context length is an architectural limit: the number of tokens the model will accept before the API rejects the request. Effective context is a behavioural property: the number of tokens over which the model still performs the task well.

These are not the same number and the gap between them is usually large. Treating the advertised figure as the usable figure is the most common reason a long-context pipeline works in a demo and degrades in production.

Why the advertised number is real but incomplete

The advertised limit is genuine. If a model states a 1M window, you can send close to a million tokens and get a response. Kimi K3, GLM-5.2, both DeepSeek V4 variants and MiniMax M3 all ship 1M windows; Kimi K2.6 ships 256K.

What the number does not tell you is how attention quality behaves across that span. Every token can attend to every earlier token, but attention is a weighted average, and as the number of candidates grows, the weight available to any single relevant token shrinks. Retrieval from a long context is a competition, and competition gets harder as the field widens.

So the limit describes what the model will accept, not what it will reliably use. Context windows explained covers the mechanics of the window itself; this article is about the shortfall.

What effective context actually means

Effective context is task-relative. There is no single number for a model, only a number for a model on a task at an accuracy threshold you have chosen.

Simple retrieval — find one distinctive fact in a pile of unrelated text — usually holds up much further than synthesis. Asking the model to reconcile three facts scattered across a long document is harder, because it must locate all three and hold them simultaneously. Asking it to notice something absent is harder still, since there is no token to attend to.

This is why a model can post excellent results on a lookup benchmark and still fail on your 400K-token codebase question. The benchmark measured one point on a curve you care about elsewhere. Needle-in-a-haystack tests covers what those benchmarks do and do not establish.

Tokenization moves the number too

Effective context is measured in tokens, but you care about content. How much of your content fits depends on how well the model tokenizes it.

A tokenizer trained on an English-heavy corpus fragments other scripts, and a codebase in an unusual language or with long unique identifiers compresses worse than plain prose. The same document can be materially more tokens on one model than another, so two models with identical advertised windows hold different amounts of your actual data.

Measure this before choosing on window size. Tokenizer differences across models covers how to run the comparison on your own content rather than trusting a rule of thumb.

Position matters, not just volume

Where information sits in the prompt affects whether it is used. Material at the very start and the very end is attended to more reliably than material in the middle, an effect robust enough to have its own name.

The practical consequence is that effective context is not a simple prefix. You cannot assume the first 200K tokens work and everything after degrades. A fact buried at the 60 percent mark of a 300K prompt can be missed while one at the 95 percent mark is found.

If a specific document matters most to the answer, place it near the end of the prompt rather than trusting the model to weigh it fairly from the middle. The lost-in-the-middle problem covers the effect and the mitigations.

Cost and latency scale before quality fails

Even where a long prompt works, it is not free. Input tokens are billed on every request, and a large fixed context resent on each turn of a conversation multiplies quickly.

Latency scales too. Time to first token grows with prompt length because the whole input must be processed before generation starts. A pipeline that feels responsive at 20K tokens can feel sluggish at 300K even when the answer is correct.

Prompt caching changes the arithmetic substantially when the long portion is stable across requests, which is the common shape for a system prompt plus a fixed document set. Kimi K3 prices cached input at $0.30 per million against $3.00 uncached. Prompt caching explained covers when the cache actually hits.

How to measure your own effective context

Build the test from your real workload rather than a synthetic one. Take genuine documents, genuine questions, and grade against genuine answers.

Run the same question set at several context sizes — say 10K, 50K, 150K, 400K — by padding with real but irrelevant material from the same corpus. Plot accuracy against size. The point where accuracy falls below your threshold is your effective context for that task, and it is the only number that should drive design decisions.

Repeat this per task type. Retrieval, multi-hop synthesis and absence detection will give you three different curves, and the lowest one governs any pipeline that mixes them. How to benchmark LLMs on your own work covers building the harness.

The decision rule

Use long context when the material genuinely needs joint reasoning and you have measured that the model holds up at that size. Use retrieval when the answer depends on a small, findable subset of a large corpus.

The failure mode to avoid is filling the window because it is available. A model given 500K tokens where 20K would do is slower, more expensive and often less accurate than the focused version. RAG versus long context covers where the boundary sits in practice.