Parallel Tool Calls: When Concurrency Helps an Agent

Most tool-calling APIs let a model emit several calls in one response, and most agent loops execute them one after another anyway. Running them concurrently is a small change to the executor and frequently the largest latency win available.

It is also the change most likely to introduce a race condition into a system that previously had none. The useful framing is not whether to parallelise but which operations are safe to, and what the loop should do when only some of them succeed.

What concurrency actually buys

Nothing about token cost. The model still reads every result, the transcript still grows by the same amount, and the bill is identical. Parallelism buys wall-clock time and nothing else.

That is worth having when tool calls are slow relative to inference. Four file reads at 50ms each are not worth the complexity. Four HTTP requests to services that take two seconds each are a six-second saving on a single turn, repeated across every turn that fans out.

Measure before you build it. If your traces show the model waiting on inference rather than on tools, concurrency in the executor is optimising the wrong half of the loop. Agent observability and tracing covers getting the per-phase timings that tell you which half you are in.

Reads parallelise, writes do not

The safe default is a simple partition. Pure reads — file reads, searches, lookups, GET requests — can run concurrently without coordination because none of them observe each other's effects.

Anything that mutates state should run serially, in the order the model emitted it. Two concurrent edits to the same file produce a result that depends on scheduler timing, which is the exact class of bug that reproduces once a week and never in a test.

A practical implementation marks each tool as read-only or mutating in its definition and lets the executor group accordingly: run the read-only calls in a batch, then the mutating ones one at a time. That gives most of the benefit with almost none of the risk.

Dependencies the model did not declare

Even among reads, the model sometimes emits calls where the second only makes sense given the first. It may search for a symbol and read a file it expects to contain it, in the same response.

Nothing in the tool-calling protocol expresses that dependency, so a naive executor runs both concurrently and the second one operates on a guess. It usually still works, which is what makes the failure mode hard to spot.

You cannot fix this from the executor alone. The mitigation is on the prompt side: instruct the model to emit only independent calls together, and keep dependent steps on separate turns. That trade — one extra turn for correctness — is almost always worth taking.

Partial failure is the hard part

With serial execution, a failure stops the batch and the model sees exactly what happened. With concurrency, three calls succeed and one fails, and the loop has to decide what to send back.

Return everything, clearly labelled. Each result should be attributed to the call that produced it, with failures reported in the same shape as successes rather than as an exception that discards the batch. The model can then act on what worked and retry only what did not.

Never let one failure suppress the successful results — that throws away work you already paid for and invites the model to redo it. Tool call retries covers deciding which of those failures is worth retrying at all.

Attribution and ordering in the transcript

Results have to be matched to their calls by identifier, not by position. If your executor returns results in completion order without the call ID attached, the model will occasionally associate the wrong output with the wrong request, and the resulting confusion looks like a hallucination.

Most tool-calling APIs give each call an ID for exactly this reason. Use it, and consider re-ordering results into the order the calls were emitted before appending them, so the transcript reads deterministically regardless of which finished first.

Determinism here also makes prompt caching viable. A transcript whose ordering varies run to run invalidates the cache prefix on every turn, which quietly costs far more than the concurrency saved. Prompt caching covers why prefix stability matters so much.

Bound the fan-out

A model that can emit calls in parallel will sometimes emit a lot of them. Twenty concurrent reads against a local filesystem is fine; twenty concurrent requests against a rate-limited API is a 429 storm that costs you the whole turn.

Cap the concurrency per tool, and cap the total number of calls accepted in one response. When the model exceeds the cap, execute up to the limit and tell it plainly that the rest were not run and can be re-requested. Silently dropping them is much worse than a legible refusal.

The cap also protects your sandbox. Concurrency multiplies whatever blast radius a single call has, so the isolation you rely on needs to hold under simultaneous access. Agent sandboxing covers what that isolation should look like.

When to leave it serial

Keep execution serial when tools are fast, when almost everything you expose mutates state, or when you are still debugging the agent. Serial execution produces traces you can read top to bottom, and that readability is worth more than latency while the behaviour is still unstable.

Add concurrency once the loop is stable, once your traces show tool time dominating, and once each tool is honestly labelled read-only or mutating. In that order — the labelling is the part that makes the rest safe, and it is the part teams skip.

The overall shape to aim for: parallel reads, serial writes, capped fan-out, results attributed by ID and returned in emission order, partial failures reported rather than swallowed. That covers nearly every case without needing a scheduler. Agent loop anatomy covers where this sits in the wider loop.