Agent Checkpointing: Saving Work a Long Session Can Lose

A forty-turn agent session that dies on turn thirty-eight has cost you the entire run. Not just the tokens — the tool calls, the wall-clock time, and whatever partial changes it left in the working tree that someone now has to reason about.

Checkpointing is the unglamorous fix. Save enough state at the right boundaries that a failed run resumes rather than restarts. The hard part is not the saving; it is knowing what constitutes state when part of your state lives outside the process.

What actually needs saving

Three separate things, and teams routinely save one and assume they have all three.

The conversation is the model's entire memory: system prompt, every message, every tool call and result. Without it the resumed agent knows nothing about what it already tried. It is also the largest and most expensive piece, and the one worth compacting before you store it.

The workspace is whatever the agent has changed on disk. In a git repository this is nearly free — a commit or a stash gives you a restorable point with a name. Outside one you need an explicit snapshot, and it needs to be cheap enough to take often.

The external effects are everything the agent did that leaves your machine: API calls, messages sent, records created. These cannot be snapshotted and cannot be rolled back, and they are what makes naive checkpointing dangerous rather than merely incomplete.

Boundaries: checkpoint on meaning, not on ticks

Checkpointing every turn is simple and mostly wasteful. Many turns produce no durable change — a read, a search, a reasoning step — and a snapshot of them buys nothing but storage.

Checkpoint after a verified state change instead. A file written and the test suite still passing is a point worth returning to. A file written mid-edit is not, and resuming from it is worse than resuming from before it.

If your agent has explicit phases, the phase transitions are the natural boundaries: they have a defined set of completed work behind them and a clean entry condition ahead. Agent state machines covers building those boundaries deliberately.

The side-effect problem

You can restore a filesystem. You cannot unsend an email, un-post a comment or un-charge a card. Any checkpoint-and-resume design that ignores this will eventually replay an irreversible action.

The mitigation has two halves. First, record every external effect in the checkpoint as an event log — what was called, with what arguments, and what came back. Second, make the tools that perform them idempotent or duplicate-aware, so a replayed call recognises itself and returns the original result rather than acting again.

Where neither is possible, mark the tool as unreplayable and have resumption stop and ask a human before crossing it. A pause that requires a person is a much better outcome than a duplicate deployment nobody noticed. Human-in-the-loop design covers structuring those interrupts.

Compact before you store

Storing the raw transcript at every checkpoint is the obvious implementation and it scales badly, because the transcript is the thing that grows fastest in a long session.

Store a compacted form instead: the system prompt verbatim, a running summary of what has been established, the last few turns in full, and stubs for superseded tool results recording that the call happened and what it concluded. That is usually a fraction of the size and loses very little that matters.

It also improves the resumed run. An agent restored from a compacted transcript starts with a cleaner context than the one that died, which frequently makes it perform better than the original would have. Context compaction strategies and summarisation in agent loops cover doing this without dropping constraints.

Metadata is what makes a checkpoint usable

A blob of state you cannot reason about is not much use at 3am. Every checkpoint should carry the task identifier, the turn number, the phase, the cumulative token and currency spend, the git revision if there is one, and a one-line human-readable note about what had been achieved.

That note is disproportionately valuable. When someone is deciding whether to resume, discard or hand a run to a person, "tests failing on two of six cases, patch applied to auth middleware" answers the question instantly. A turn counter does not.

Include the spend figures because they determine whether resuming is even sensible. A run that has already consumed its budget should not silently resume and consume another. Agent token budgets covers carrying budget across a resume.

Retention and cost

Checkpoints are cheap individually and expensive in aggregate, particularly if you keep every transcript for every run. Decide the policy before you turn it on rather than after storage growth becomes a ticket.

A workable default: keep the last few checkpoints of a running session for resumption, keep the final checkpoint of every completed run for auditing, and expire intermediate ones on a short clock. Failures are worth keeping longer than successes, because those are the ones you will want to inspect.

Storing checkpoints outside the agent process is what makes them useful for anything beyond a crash. A checkpoint on the machine that died is not a recovery mechanism. Long-running agents covers the operational side.

Test it by killing the process

Checkpointing is one of those features that appears to work until the first real failure. The only honest test is to kill the agent at a random turn, resume from the last checkpoint, and check that the outcome matches an uninterrupted run.

Do it repeatedly at different points. The bugs cluster around partially completed operations — a file half written, a tool call issued but whose result never arrived — and those only appear if you interrupt at exactly the wrong moment, which random killing eventually does.

Once that passes, the resumption logic itself is the next thing to get right, and it is a separate problem with its own failure modes. Agent resumption patterns covers restarting from a checkpoint safely.