The Token Cost of Long Context: When 1M Is Worth Filling

Kimi K3, GLM-5.2, both DeepSeek V4 variants and MiniMax M3 all ship 1M-token context windows. The capability is genuinely useful and it is routinely misused, because a window being large enough is not a reason to fill it.

What filling it actually costs

Work the arithmetic across tiers, using 800,000 input tokens as a large but realistic prompt.

On DeepSeek V4 Flash at around $0.14 per million, that is roughly $0.11. On DeepSeek V4 Pro at around $0.44, about $0.35. On GLM-5.2 at around $1.40, about $1.12. On Kimi K3 at $3, about $2.40 — for a single request, before any output.

That spread is the whole strategic point. At the budget end, reading enormous inputs is close to free and genuinely changes what architectures are sensible. At the frontier end, a single large prompt costs more than most complete agent runs, and doing it repeatedly is a serious line item.

Repetition is what turns it expensive

One large prompt is affordable at almost any tier. The problem is that large prompts rarely happen once.

In an agent loop the context is resent every turn, so an 800,000-token prompt across twenty turns is 16 million input tokens — at Kimi K3 rates, roughly $48 for a single task. In a question-answering flow, ten questions against the same document means ten full prompts unless caching intervenes.

This is where prompt caching stops being an optimisation and becomes structural. Kimi K3's cached input rate of $0.30 against $3 turns that $48 into something closer to $5 if the document sits in a stable prefix. Prompt caching savings math covers the arrangement.

You are also paying for attention you do not get

The cost argument is only half of it. Usable attention degrades across long inputs on every model — content in the middle of a very long prompt is attended to least reliably, regardless of the advertised maximum.

So a 800,000-token prompt is not merely expensive. It is expensive and gives worse results than a well-chosen 30,000-token prompt containing the same relevant material, because the relevant material is no longer competing with 770,000 tokens of noise.

That combination — worse and dearer — is unusual. Most engineering trade-offs let you buy quality with money. This one does not. The lost-in-the-middle problem covers the mechanism.

When filling the window is right

Three cases genuinely justify it.

The relevant material cannot be identified in advance. If the question is "is there anything anywhere in this codebase that does X", retrieval has to guess what to retrieve and can miss. Reading everything is the correct approach, and on a cheap model it is affordable.

Reasoning spans the whole input. Summarising a long document, finding inconsistencies across a corpus, or tracing a data flow through many files needs the whole thing present. Retrieval fragments it.

Building the retrieval pipeline costs more than the tokens. For a small or medium codebase queried occasionally, an index to build, chunk, embed, maintain and keep fresh is real engineering. Feeding the whole thing to DeepSeek V4 Flash for pennies is often the better trade.

When retrieval wins

When the corpus is large enough that even the cheap tier gets expensive at volume. When it changes constantly, so caching never warms. When access control matters and different users must see different subsets. And when latency matters, because prefill on a huge prompt has a floor no model beats.

RAG vs long context works through the comparison properly rather than as an ideological position.

The decision rule

Estimate three numbers: how many tokens the full context is, how many times you will send it, and whether a stable prefix makes it cacheable.

Large context sent once, on a cheap model, is fine. Large context sent repeatedly without caching is the expensive mistake. Large context sent repeatedly with caching is usually acceptable. And large context on a frontier model, uncached, in a loop, is the configuration that produces a bill nobody can explain.

Measure your actual cached-input fraction before assuming caching is protecting you. It frequently is not, and the usual cause is a variable element near the top of the prompt invalidating everything after it.

Output length is billed separately and often higher

Long-context discussion focuses almost entirely on input, because that is where the volume is. Output rates are typically several times higher per token, and on generative workloads that reverses the picture.

Kimi K3 lists output at $15 per million against $3 for input — a five-fold difference. A task that reads 50,000 tokens and writes 30,000 spends more on the writing than the reading, even though the input looks larger in isolation.

GLM-5.2 is worth noting here because of its 128K maximum output, which is unusually high. That capability is genuinely useful for migrations and bulk generation, and it is also a way to spend a lot of money quickly if a prompt accidentally invites a very long response. Cap output length explicitly rather than relying on the model to stop.

The practical rule is to budget input and output separately rather than reasoning about total tokens, because the rates differ enough that a combined figure hides which side is driving the bill.

Measure before optimising

Log input tokens, cached input tokens and output tokens per request, then look at the split across a representative week. Teams are frequently wrong about which of the three dominates their own workload, and optimising the wrong one produces disappointing results after real effort.