Context Bloat: Paying Repeatedly for Tokens Nobody Reads

Context bloat is the slow accumulation of prompt content that nobody removes. A system prompt grows an instruction at a time. A tool returns more than it needs to. Few-shot examples get added and never pruned.

None of it is individually significant. Collectively it is often the largest controllable line in an inference bill, and unlike most optimisations it degrades quality at the same time as it costs money.

The multiplier that makes it expensive

A token added to a system prompt is not paid once. It is paid on every request for as long as it stays there.

Two thousand tokens of accumulated instructions across a hundred thousand monthly requests is 200 million input tokens. At DeepSeek V4 Pro rates around $0.44 per million, that is roughly $88 a month for text nobody has read since it was added. At Kimi K3 rates it is closer to $600.

Inside an agent loop the multiplier compounds again, because the transcript is resent every turn. Content added early is paid for on every subsequent step of the run. Cost per agent run covers that arithmetic.

Where bloat accumulates

System prompts. The classic case. Each incident produces a new instruction — "never do X", "always check Y" — and none are ever removed, because removing one feels risky and nobody knows which ones still matter. After a year the prompt is a changelog of past failures.

Tool output. A search that returns full file contents rather than matching lines. A test runner that returns the whole log rather than failures. These are the largest single contributors in agent workloads and the easiest to fix.

Few-shot examples. Added to fix a specific behaviour, kept indefinitely. Often the model no longer needs them, and occasionally they now conflict with instructions added later.

Retrieved context. A retrieval step configured to return the top twenty chunks when the top five would do. Nobody revisits the parameter after it works.

Conversation history. Chat interfaces that resend the entire session rather than a managed window.

It costs quality as well as money

This is the part that makes bloat worth fixing even when the money is not the issue.

Attention is finite and normalised across positions. Adding irrelevant material dilutes the weight available for relevant material, and content in the middle of a long prompt is attended to least reliably. A critical instruction sitting at position 4,000 of a 40,000-token prompt is in the weakest possible position.

So the usual symptom of a bloated prompt is not just a larger bill. It is instructions being followed inconsistently, and the standard response — adding another instruction restating the first more forcefully — makes the underlying problem worse. The lost-in-the-middle problem covers the mechanism.

Auditing it

Start by measuring. Log input token counts broken down by component: system prompt, tool definitions, retrieved context, history. Most teams have never seen this breakdown and are surprised by which part dominates.

Then test removal rather than reasoning about it. Take your evaluation set, remove one section of the system prompt, and re-run. If results are unchanged, that section was costing money for nothing. Working through a long prompt this way typically removes a third of it without any measurable quality loss.

The reason this works is that many instructions were added to fix a problem in a model that has since been upgraded, or to fix a problem that was actually caused by something else. They were never load-bearing.

Keeping it from returning

Bloat is a process problem more than an engineering one, so the durable fixes are process fixes.

Date instructions when you add them, with a note of what they were for. An undated line nobody understands is one nobody will dare delete. Review the prompt on a schedule rather than only when something breaks. And require that a new instruction come with an evaluation case demonstrating it is needed — which both justifies its existence and gives the next person a way to test removing it.

For tool output, set a hard token ceiling per tool and truncate with a note rather than returning everything. A truncated result the model can ask to expand is better than a complete one that crowds out everything else. Tool result formatting covers the patterns.