Context Compaction: Keeping Agents Alive Past Turn Twenty

An agent's conversation grows every turn. Tool output accumulates, reasoning accumulates, and by turn twenty the transcript can dominate the context window with material that stopped being relevant fifteen turns ago.

Two things go wrong simultaneously. Cost rises, because the whole transcript is resent each turn. And quality falls, because relevant information is now buried among irrelevant information in the weakly attended middle of a very long prompt.

Why a bigger window is not the answer

The instinct is to reach for a 1M-context model and stop worrying. That helps less than expected.

Usable attention degrades across long inputs regardless of the advertised maximum, so a transcript that has grown to 400,000 tokens is not being read evenly. The important instruction from turn three is competing with forty file dumps.

And cost still scales with what you send. A large window makes an unmanaged transcript possible, not affordable. The lost-in-the-middle problem covers the attention side.

What actually needs to persist

Sort the transcript into categories, because they have different lifetimes.

Never drop: the system prompt, the original task, and any constraint that must hold throughout. These are small and load-bearing.

Summarise: completed sub-tasks. Once a file has been found and read and the relevant function identified, the full contents can go and a one-line note that the function lives at a given path and does a given thing can stay.

Drop entirely: raw tool output that has been acted on, failed attempts that were superseded, and exploratory reads that led nowhere. This is usually the bulk of the transcript by volume.

Keep verbatim: the last few turns. Recent context is what the model is actively reasoning over, and summarising it breaks continuity.

Three strategies, in increasing sophistication

Sliding window. Keep the system prompt, the task, and the last N turns. Drop the rest. Trivial to implement and surprisingly effective for short-horizon work, but it loses information that turns out to matter later.

Summarise on threshold. When the transcript exceeds a token budget, ask the model to summarise everything except the recent turns, then replace that span with the summary. This preserves more and costs an extra call at each compaction.

Structured state. Maintain an explicit state object outside the conversation — files examined, facts established, decisions made, current plan — and rebuild the prompt from that state plus recent turns. More engineering, considerably more control, and much better behaviour on long runs.

Most teams should start with the second and move to the third when they find themselves debugging lost information.

The failure mode to design against

Compaction loses things, and the losses are silent. The model does not know something was removed; it simply proceeds without it.

The characteristic symptom is an agent re-doing work — reading a file it already read, re-deriving a conclusion it already reached, or violating a constraint it was given before the last compaction.

Two defences. Keep an append-only list of established facts that compaction never touches, separate from the conversational flow. And after compacting, include an explicit note that earlier context was summarised, so the model knows to ask rather than assume when something is missing.

Compaction interacts badly with caching

This is worth knowing before you tune either.

Prompt caching reuses a matching prefix. Compaction rewrites the middle of the transcript, which changes the prefix from that point forward and invalidates the cache for everything after it.

So a compaction event costs a full uncached prefill on the next turn. Compacting frequently in small increments can therefore cost more than compacting rarely in large ones, even though it keeps the transcript smaller.

The practical resolution is to compact on a threshold rather than continuously, and to put the compacted summary as early in the prompt as possible so the stable region below it stays cacheable. Prompt caching explained covers the prefix rules.

Measure before tuning

Instrument three things: transcript token count per turn, the turn at which compaction triggers, and the rate at which the agent repeats work it already did.

That third metric is the one that tells you whether compaction is too aggressive. If repeated work rises after a compaction change, you removed something load-bearing. If it stays flat while token counts fall, the compaction is working.

Without that measurement, compaction tuning is guesswork, and the failure is invisible because it looks like the agent being slightly stupid rather than like a bug you introduced.