Tracing Multi-Step Agents: Building a Trace Tree That Holds

The first tracing setup most teams build produces a flat stream of model calls with timestamps. It looks like progress until the first real incident, when you discover you cannot tell which tool result caused which decision, which of the six parallel searches came back empty, or whether the subagent that hung was even part of this run.

Multi-step agents need a tree, and the difficulty is not drawing the tree. It is keeping it connected across the places where the parent and the child do not share a call stack.

The shape you are aiming for

One root span per task, lasting the entire run. Under it, a child span for every model call and a child span for every tool invocation, in the order they happened. Under any tool that delegates to another agent, that agent's own root span nested as a child rather than floating as a separate trace.

Depth matters more than it looks. A flat list of sibling spans tells you the order of events; a tree tells you the causality. When step fourteen goes wrong, you want to collapse the run to its top level, find the branch that took eleven seconds and four retries, and expand only that.

The OpenTelemetry GenAI conventions describe this shape directly — an agent invocation span with chat spans for model calls and tool execution spans beneath. Adopt the shape even if you write the exporter yourself. The attribute names in those conventions are still marked as in development, so keep a thin mapping layer rather than hardcoding names through your codebase.

Propagation is where trees become forests

Inside a single synchronous function, most tracing libraries handle parenting for you through implicit context. Agents break that assumption constantly, and each break produces an orphaned trace that looks like an unrelated run.

The usual culprits are the same three. A subagent launched as a separate process or worker, where the parent context never crosses the boundary. A tool that enqueues work and returns immediately, so the actual execution happens minutes later with no ambient context. And a retry or fallback path implemented in a wrapper that starts its own trace.

The fix in every case is to serialise the trace context — trace identifier, current span identifier, sampling decision — and carry it explicitly through whatever boundary you are crossing. Put it in the job payload, the subprocess environment, the message header. Then restore it as the parent on the other side.

Where genuine asynchrony makes strict parenting wrong — a background task the parent does not wait for — use a span link instead of a parent relationship. A link records the association without claiming the child ran inside the parent's lifetime, which keeps your duration arithmetic honest. Delegated work is common enough that this is worth getting right early; subagents and delegation covers when that structure is appropriate at all.

Fan-out and parallel tool calls

Models increasingly emit several tool calls in one turn, and harnesses execute them concurrently. Traced naively, you get overlapping siblings with no indication that they belong to the same decision.

Wrap the batch. Open one span for the turn's tool execution, nest each concurrent call underneath it, and record on the parent how many calls were requested and how many succeeded. Now the collapsed view shows one entry per turn, the wall-clock cost of the batch is the parent duration rather than a sum you have to compute, and a single failing call among six is visible without expanding everything.

Record the tool call identifier the model emitted on each child span. When a result gets attached to the wrong call — which happens more often than you would like in hand-rolled harnesses — that identifier is the only thing that shows it.

Retries, fallbacks and what was actually served

A retry is a separate attempt and deserves a separate span. Folding retries into one span destroys the two facts you most want: how many attempts there were, and whether the successful one differed from the failures.

Nest attempt spans under a logical operation span, number them, and record the failure reason on each. Then put the outcome on the parent. This makes retry storms visible as visual width rather than as a number buried in an attribute.

Always record the model identifier the provider actually served, not the alias you asked for. Routing layers, fallbacks and floating aliases all make those diverge, and a trace that records your intent rather than reality will send you debugging the wrong model. Model routing and fallbacks covers why that divergence is normal in production.

Streaming needs more than a duration

A streamed model call has two latencies that behave differently and have different causes. Time to first token reflects queueing and prompt processing; the remainder reflects generation speed and output length.

Record both. Set an event on the span when the first token arrives, and keep the span open until the stream closes. A single duration averages the two into a number that moves for reasons you cannot distinguish.

Also record how the stream ended. A connection dropped mid-generation and a clean stop look identical downstream unless the finish reason is captured, and truncated output is a common cause of otherwise inexplicable agent behaviour. Streaming and server-sent events covers the failure modes on the transport side.

Annotate what the harness decides

The spans above capture what the model and the tools did. They miss what your own code did between them, and that gap is responsible for a surprising share of confusing traces.

When your harness truncates the transcript, drops an old tool result, injects a reminder, rewrites a tool schema, blocks a call or forces a different model, emit an event on the current span saying so. Without it you get a trace where step nine lacks information step eight clearly had, and nothing to explain the disappearance.

These annotations are also what let you correlate behaviour with configuration. Recording the prompt version and the compaction policy on the root span means your observability metrics can be sliced by them later.

A good trace is a replayable run

If you capture tool arguments and tool results in full, you have accidentally built something more valuable than a debugging aid. The recorded results make the run deterministic on replay even though the original was not, because the nondeterministic world has been frozen into fixtures.

That gives you two things. A minimal reproduction: trim the trace to the last few steps before the failure, replay from there, and iterate in seconds instead of minutes. And a regression corpus: promote interesting failed runs into a suite that replays against every prompt or model change, which is exactly the input prompt regression testing needs and is far more representative than anything written by hand.

Start with the tree and the propagation. Everything else on this list is an attribute you can add later, but a trace that fragments at the first subagent boundary cannot be repaired after the fact — and it is the runs that cross those boundaries that you will most need to debug when an agent gets stuck.