Measuring Agent Success Rate Without Fooling Yourself

Someone asks how well your agent works and you say eighty-seven percent. The number is confident, specific, and almost certainly not measuring what the person asking has in mind.

Success rate is a ratio, and every ratio is defined by its denominator. Change what counts as an attempt, what counts as success, and which tasks are in the pool, and the same system reports anywhere from forty to ninety-five percent. None of those numbers are lies. They are answers to different questions.

Define the denominator first

Start with the unit. Is an attempt a single user request, a full session, or a task that may span several sessions with a human nudging in between? A per-request rate flatters you because most requests are easy follow-ups. A per-task rate is harsher and far more useful.

Then decide what happens to abandoned runs. If a user gives up halfway and closes the tab, that attempt did not fail on any assertion you wrote, but it certainly did not succeed. Excluding abandonments from the denominator is the single most common way teams accidentally inflate the number.

Decide too whether escalations count as failures. An agent that correctly recognises it lacks the information to proceed and asks a human is behaving well. Counting that as a failure punishes exactly the behaviour you want, so track escalation as its own category rather than folding it into either bucket. Human-in-the-loop design covers when escalation is the right outcome.

Define success programmatically or not at all

If you cannot write a check that decides success without a human reading the transcript, the task is underspecified and the resulting number will drift with whoever is grading.

The strongest checks compare end state against an expected state: the test suite passes, the file matches, the row exists with the right values, the external API received a call with the right arguments. These are cheap, deterministic and immune to rewording.

Where the output is genuinely prose, replace the holistic score with a set of narrow binary questions. Did it cite a source for every factual claim. Did it stay within the stated scope. Did it avoid asserting a capability the system does not have. Binary questions can be answered consistently by a judge model or a human; a one-to-ten quality score cannot, and it clusters around seven regardless of what happened.

One number is never enough

Report success rate at a single attempt alongside consistency across repeated attempts. An agent that succeeds ninety percent of the time on any given try is not ninety percent reliable when chained across an eight-step workflow, because the per-step rates multiply and you land near forty-three percent end to end.

This is why the strict all-attempts-succeeded metric matters for anything unattended. If nobody is picking the good run out of five, the probability that every run is good is the number that describes your experience. Evaluating agent reliability works through the arithmetic in detail.

Report cost and steps alongside success, per completed task rather than per request. An agent that halves its per-call cost while dropping ten points of success rate got more expensive, because you now pay for the failures and the retries. Agent cost control patterns covers keeping that ratio visible.

Segment before you average

A single blended rate hides everything actionable. The interesting question is never how well the agent does overall; it is which slice of work it fails on.

Segment by task type, by input size, by whether the task requires a write action, and by which tools are involved. A system at eighty-seven percent overall might be at ninety-six on read-only queries and fifty-two on anything that mutates state, which is a completely different engineering problem to the one the blended number suggests.

Segment by repository or customer too, if your agent touches more than one. Failures cluster. A codebase with unusual build tooling or a customer with an odd data shape will often account for most of the tail, and fixing that one case moves the aggregate more than any prompt work.

Watch for the metrics that quietly go up

Success rate can improve for reasons you would not endorse if you saw them stated plainly.

An agent that learns to attempt less will succeed more often at what it does attempt. If refusal rate and success rate rise together, you have traded coverage for the metric. Track attempted-and-completed as a share of all requests, not just of attempted ones.

Similarly, a suite that ages will drift upward as the easy cases stay and the hard ones get quietly retired for being flaky. Keep a frozen core set that never changes so you have a stable baseline, and version the suite so you can tell whether a jump came from the system or from the test set. Agent regression suites covers keeping that gate honest.

Public benchmarks answer a different question

Leaderboard numbers are real measurements of real things, but the thing they measure is a model on a curated task distribution with a fixed harness. Your agent is a model plus your tools, your prompts, your data and your users.

Harness differences alone move published figures substantially, which is part of why two credible sources report different scores for the same model. Why benchmarks disagree and what SWE-bench actually measures are worth reading before you use any published rate as a forecast of your own.

Use benchmarks to shortlist candidates. Use your own measured success rate to decide. The two rarely rank models identically, and when they conflict, yours is the one describing your production traffic.

A definition worth writing down

Pick one unit of work and stick to it. Count abandonments as failures and escalations as their own category. Define success as a programmatic end-state check. Run each case five times and report both the single-attempt rate and the all-five rate. Segment by task type and by read versus write. Publish cost and median steps per completed task next to the rate.

Write that definition down in the same place as the number. A success rate without its definition is a rumour, and it will be quoted back at you in a planning meeting six months from now with none of the caveats attached.