Partial Failure Recovery: When Half the Agents Succeed

Six workers were dispatched. Four returned useful results, one timed out, one returned something malformed. What should the orchestrator do?

Most systems answer this by accident. The code awaits everything, one promise rejects, an exception propagates, and the whole run dies — discarding four completed investigations you already paid for. Or the opposite: the failures are swallowed, the synthesis proceeds on two-thirds of the evidence, and nothing in the output admits that a third of the search never happened.

Both are decisions. Neither was made deliberately.

Partial failure is the normal case

At any meaningful fan-out width, some fraction of workers will fail on any given run. Timeouts, rate limits, tools that are briefly unavailable, workers that exhaust their step budget without concluding — none of these are rare, and their probability compounds with width.

That reframes the problem. You are not designing for an exceptional case, you are designing the ordinary control flow. A system whose only response to a failed worker is to abort will abort constantly at scale, and the wider you fan out the worse it gets, which is one of the ceilings discussed in agent fan-out limits.

Classify the failure before deciding

The right response depends on why the worker failed, and the categories are easy to distinguish if you record them.

Transient: a rate limit, a timeout, a 503. The work is probably fine on a second attempt, so retry with backoff and treat it as recoverable.

Deterministic: a malformed brief, a missing permission, a tool that will fail identically every time. Retrying is a waste; either fix the input or record the gap.

Exhausted: the worker hit its step or token budget without concluding. Ambiguous, and worth capturing the partial state — a worker that read thirty files and ran out of budget still learned something worth returning.

Degenerate: the worker returned something, but it is empty, off-topic or plainly wrong. The most dangerous, because it looks like success to any code checking only for exceptions.

Recording the class costs one field and turns recovery from a guess into a lookup.

Quorum: how much is enough

Decide in advance what fraction of workers must succeed for the result to be trustworthy, and make it explicit rather than implicit in the exception handling.

For genuinely independent exploration, a simple fraction is often adequate — four of six is a reasonable sample of a search space, provided the answer does not hinge on the missing two. For decomposed work where each worker owns a distinct necessary piece, quorum is all of them, because a missing piece is a hole in the output rather than a smaller sample.

Weight by importance where the workers are not equivalent. If one worker was auditing the security-sensitive module and five were checking documentation, losing that one matters more than losing the five, and a flat count says otherwise.

Write the rule down next to the fan-out. A quorum policy buried in a try-block is a policy nobody will find when the behaviour surprises them.

Degrade honestly

When you proceed with partial results, the output must say so. This is the part teams skip, and it is what turns a survivable failure into a misleading answer.

Carry the coverage forward into the synthesis: which areas were investigated, which were not, and why. An orchestrator that knows two workers failed can hedge its conclusions appropriately; one that receives only the four successful results will speak with the confidence of complete coverage.

Surface it to the consumer too, whether that is a human or another agent. A report that says four of six modules were audited and names the two that were not is genuinely useful. The same report without that caveat is a hazard, especially when it feeds a downstream agent that has no way to know what is missing — the cascade pattern described in agent failure modes.

Retry the piece, not the run

The expensive mistake is retrying everything. Re-running six workers because one failed costs six times what it should and discards results you already have.

Make workers individually addressable and their results individually persistable. Store each result as it arrives, keyed by worker identity and brief, so a retry re-runs exactly one brief against exactly one gap.

Give retries a budget of their own, separate from the original run, and cap it. Without a cap, a systematically failing worker will consume more than the entire original fan-out while producing nothing, and the spend shows up as the kind of surprise covered in why agent costs are unpredictable.

Consider reformulating on retry rather than repeating verbatim. A worker that exhausted its budget on a brief that was too broad will exhaust it again; a narrower brief has a better chance than a second identical attempt.

Recovering side effects

Read-only workers make all of this comfortable, because a failed worker leaves nothing behind. Workers that write are a different problem entirely.

If three of five workers made changes and two failed, the system is in a state no one designed. Whether you can proceed depends on whether the completed changes are independently valid or only make sense as a set.

The tools here are the usual ones: make every mutating action safe to repeat, so a retry cannot double-apply, and have a compensating action for anything that must be undone. Those are covered in idempotency in agent actions and rollback strategies for agents. The design rule that avoids most of it is simpler: keep the fan-out read-only and let one agent own the writes.

What to build

Never await everything with a single failure aborting the batch. Collect results and failures side by side, classify each failure, apply an explicit quorum rule, and persist results as they arrive so retries are surgical.

Then make coverage a first-class field in the output, so partial results are always labelled as partial. A system that can say what it did not check is more useful than one that quietly checks less than it claims, and the difference only becomes visible during an incident — which is exactly when you cannot fix it.