Replaying Agent Traces: Debugging a Run You Cannot Reproduce

An agent did something wrong in production. You run the same task again and it behaves correctly. This is the normal case, not the unlucky one, and it means the stored trace of the original run is the only evidence you will ever have.

Whether that evidence is usable is decided before the incident, by what the harness recorded. Most agent instrumentation records enough to build a dashboard and not enough to reconstruct a run.

What a replayable trace contains

Start from the requirement: someone should be able to reconstruct, turn by turn, exactly what the model saw and exactly what it did.

That means the full message list sent on every turn, not just the new message — the loop resends everything, and what changed between turns is often the explanation. It means the complete raw response including any reasoning content the provider returned. It means every tool call with full arguments and the full result that went back into the context.

It also means the environment: model identifier and exact version, sampling parameters, the system prompt as sent, the tool schemas as sent, and the harness version. A trace without these can be read but not explained, because the most common cause of changed behaviour is a change in one of them. Agent audit logging covers the record format.

Reading is most of the value

Before building any replay machinery, build a viewer. Turn-by-turn, showing what was added to the context, what the model produced, what each tool returned, with token counts and cost per turn.

A surprising share of agent incidents are diagnosed by reading. The tool returned an empty result and the model invented a plausible substitute. The context was compacted and a constraint went with it. A retrieved document contained an instruction. All of these are visible on inspection and invisible in any aggregate.

Make the viewer show sizes prominently. The turn where the transcript doubled is usually the turn where the run went wrong, and it stands out visually in a way it never does in a log file. Agent observability and tracing covers the metrics alongside it.

Three kinds of replay

Deterministic replay feeds the stored responses back through the harness without calling the model at all. Nothing is regenerated, so it reproduces exactly. This is the mode for debugging your own code — parsing, state handling, context assembly, stopping rules — and it is fast and free.

Model replay resends a stored turn's message list to the model and compares the new response with the recorded one. It answers whether behaviour changed after a version bump or a prompt edit, and it will not match exactly.

Branched replay takes the trace up to a chosen turn, modifies something — the system prompt, a tool result, the model — and runs forward live. This is the mode that answers counterfactuals: would a better error message have prevented the loop.

Fidelity has limits

Be clear about what replay cannot restore, because the failure mode is a confident conclusion drawn from an unfaithful reproduction.

Tool results depend on external state. Replaying a stored result is faithful to the original run and not to the world; replaying against live tools reproduces the world and not the run. Neither is wrong, but conflating them produces bad conclusions.

Model responses are not reproducible even at temperature zero, because provider-side batching and infrastructure changes affect sampling. A replay that produces different output is not evidence of a change on your side. LLM determinism and seeds covers how far this goes.

And a model version silently updated underneath a floating alias makes replay comparisons meaningless. Recording the exact version served is what lets you tell a real regression from a provider change. Pinning model versions covers avoiding it.

Traces are the best source of test cases

A trace of a real failure is an already-specified test: a real task, a real environment, and a known bad outcome.

Converting one into a harness fixture is mostly mechanical. Take the initial state and the task, add an outcome check that the failure would violate, and add it to the suite. The result is a regression test grounded in something that genuinely happened rather than in something you imagined might.

Stored tool results also make excellent scripted responses for deterministic harness tests, since they exercise the exact malformed output that broke the parser. Agent test harnesses covers wiring them in.

Storage, sampling and privacy

Traces are large. A long agent session can hold several megabytes of transcript, and storing every run in full gets expensive at fleet scale.

Sample deliberately rather than uniformly. Keep every failed run, every run that hit a limit, every run whose cost landed in the top percentile, and a small random sample of successes for baseline comparison. That is a fraction of the volume and nearly all of the diagnostic value.

Redact at capture. Traces contain everything that entered the context, which means credentials, customer data and source code, all in one object. Filter before writing rather than at read time, and set retention short for full transcripts while keeping the structured summary long. The structured summary is also what answers cost and compliance questions months later, long after the transcripts have expired.

The minimum worth building

Store full traces for failures and outliers. Build a turn-by-turn viewer showing context growth. Add deterministic replay against stored responses, which needs no provider calls and catches most harness bugs.

Add branched replay when you start asking counterfactual questions, and not before. Reading and deterministic replay answer the majority of incidents on their own. Detecting agent loops covers the failure you will read about most often.