Speculative Agent Execution: Running Ahead of the Decision

Watch a trace of a coding agent and the striking thing is how much of the elapsed time is nothing happening. The model generates for eight seconds, a tool runs for two hundred milliseconds, then the model generates again. Compute sits idle for most of the session.

Speculative execution is the idea of using that idle time on work you probably need. Start the likely next tool call before the model has finished asking for it, and throw the result away if you guessed wrong. It is the same bet processors have made for decades, and it transfers to agents with some important caveats.

The two things you can speculate on

The first is tool execution. In many states the next call is highly predictable. After the model reads a test file, it almost always runs the tests. After it edits a source file, it almost always rebuilds. You can start that command the moment the edit completes, in parallel with the model deciding to ask for it.

The second is branch exploration. When the plan reaches a genuine fork — patch A or patch B — you can pursue both and keep the one that verifies. This is expensive but it converts a decision the model is bad at into a check your build system is good at.

Both are distinct from speculative decoding, which speeds up token generation inside the provider. That happens below your layer and you get its benefit passively. Speculative agent execution is something you build in the loop.

Only speculate on safe operations

This is the constraint that decides whether the technique is usable at all. A discarded speculation must leave no trace, which means the speculated operation has to be free of side effects the rest of the system can observe.

Safe: reading files, running a build in a scratch directory, running tests, searching a codebase, fetching a document, computing a diff. If you guess wrong, you delete the result and nothing happened.

Unsafe: anything that writes to shared state, calls a payment API, sends a message, mutates a production database, or pushes a commit. Speculating on these means occasionally performing an action nobody asked for, which is not a latency optimisation but an incident.

The clean way to enforce this is to mark every tool as speculatable or not at registration time, and have the executor refuse to speculate on anything unmarked. Default to unsafe. A tool author adding a side effect later will not remember to update a flag, so review speculatable tools whenever they change. Agent sandboxing gives you a second layer if the marking is wrong.

Predicting the next call

You do not need a model to predict the next tool call. In most agent workloads the transition distribution is sharply peaked and a simple table learned from your own traces does well.

Log every observed transition from tool call to next tool call, conditioned on the outcome. Keep the ones where a single successor accounts for a large share of the mass — after a failed test run, the next call is a file read of the failing test with high probability. Speculate only on those.

Set the threshold using economics, not intuition. If a speculation costs you a build worth a fraction of a cent and saves two seconds when right, a fifty percent hit rate is a good trade. If it costs a full agent session, you want ninety. Measure your actual hit rate per transition and prune the ones that fall below their break-even.

Argument prediction is the harder half. Predicting that the model will run the tests is easy; predicting which file it will read is not. Speculate on argument-free or argument-obvious calls first, and treat parameterised speculation as a later optimisation.

Branch speculation and its price

Running two candidate solutions in parallel and keeping the verified one is genuinely effective for tasks where a check exists. It is also a direct multiplier on cost: two branches means roughly double the tokens for that segment of work.

That trade is defensible when the alternative is a wrong answer discovered ten minutes later, or when latency is the binding constraint and the budget is not. It is indefensible as a default. Gate it on task value, and keep the branch count at two — three or more rarely improves the outcome enough to justify the spend.

Be honest about the accounting. Speculation makes cost per session less predictable, which is already the hardest thing about agent budgeting. Track speculative spend as its own line so a hit-rate regression shows up as a cost regression. Why agent costs are unpredictable covers the underlying variance, and agent cost control patterns covers the ceilings.

Speculation and caching are the same idea

A speculative result that arrives before the model asks for it is, functionally, a cache entry with a very short lifetime. It is worth wiring both through the same layer.

When the model finally issues the call, the executor checks whether a completed or in-flight speculation matches. A match returns immediately or awaits the in-flight run rather than starting a second one. A miss discards the speculation and runs normally.

Matching has to be exact on the normalised call, including arguments, or you will serve a result for a different question. Get the normalisation wrong and you have built a subtle correctness bug that looks like model hallucination. Agent caching strategies covers the keying rules that apply here too.

Whether it is worth building

Speculative execution is a latency optimisation with a real complexity cost. It only pays if latency is actually your problem, which for most batch and background agent work it is not.

It pays for interactive agents where a developer is watching, for long-running sessions where seconds accumulate into minutes, and for tool calls that are slow in absolute terms — a build that takes forty seconds is worth starting early in a way that a file read is not.

Start by measuring. Instrument your traces for time spent generating versus time spent in tools, and look at the transition table for peaked distributions. If your slowest tools are also your most predictable successors, the win is there. If not, spend the effort on reducing generation latency instead.