Agent Test Harnesses: Testing a System That Is Not Deterministic

Testing an agent with the tools you use for ordinary software fails immediately. Assertions expect one correct output; an agent produces a different valid one each run. Fix the sampling temperature and you still get variation, because the tool results feeding back into the loop are not identical either.

The working reframe is that an agent test measures a rate, not a value. The question is not whether this run succeeded but what fraction of runs succeed, and how that fraction moves when you change something.

Tasks, not assertions

The unit in an agent harness is a task: a starting workspace, a request, and a check that determines success.

The starting workspace has to be reproducible, which usually means a fixture repository at a pinned commit, restored fresh before each run. An agent that mutates its workspace contaminates every subsequent run in the same directory, and the resulting flakiness is easy to mistake for model variance.

The check is the part worth investing in. It should verify the outcome rather than the route: the tests pass, the endpoint returns the right shape, the function exists with the right behaviour. Checking that the agent edited a particular file makes the test brittle against perfectly good alternative solutions. Verifier agents covers writing outcome checks.

Run each task many times

A task that passes once tells you almost nothing. Run each one repeatedly — five times is a usable floor, more where the decision matters — and record the pass rate.

Report the rate with its uncertainty. Five successes out of five is compatible with a true rate anywhere from roughly seventy per cent upwards, which means a change from five-of-five to four-of-five is not evidence of anything.

This is what makes agent testing expensive and why suites stay small. Ten tasks at five runs each is fifty agent sessions per evaluation, which is a real cost and the reason to keep the suite focused on decisions you actually make. How to benchmark LLMs on your own work covers task selection.

Record everything, not just the verdict

A pass rate tells you something changed and nothing about why, so the harness should capture the material needed to answer the second question.

For every run: the complete transcript, every tool call with arguments and results, turn count, wall-clock duration, token usage and cost, the model identifier and version, and the failure reason where it failed.

Failure reasons want a taxonomy rather than free text. Hit a turn limit, hit a cost ceiling, produced a wrong answer, crashed on a tool error, stopped early claiming success. These have completely different fixes and merging them into one failure count hides which one you have.

Store the traces addressably so a specific failing run can be pulled up and read. Most of the value of a harness is realised when someone reads a failing transcript. Replaying agent traces covers making them navigable.

Mock the model, or mock the tools, but know which

Two kinds of test hide under the same name and they answer different questions.

A harness test runs the real model against real tools and measures end-to-end capability. It is slow, costs money, and is the only thing that tells you whether the system works.

A loop test replaces the model with a scripted sequence of responses and checks that the harness behaves: that tool errors are surfaced correctly, that the turn ceiling fires, that loop detection triggers, that a timeout produces the right message. These are fast, deterministic, free, and belong in ordinary CI.

Most agent bugs are in the second category. The harness mishandles an error, truncates the wrong thing, or fails to stop. Scripted-response tests catch those in seconds and they are consistently under-built relative to how much they find. Script the awkward cases specifically: a truncated response, a malformed tool call, a tool that returns nothing.

Control what you can

You cannot make an agent deterministic, but you can remove the variance that is not the model.

Pin the model version explicitly rather than using a floating alias, or your suite silently measures a different system after a provider update. Pin fixture commits, tool versions and dependencies. Set the sampling parameters explicitly rather than inheriting defaults.

Then treat the remaining variance as the signal it is. A task with a highly variable pass rate is telling you something about the task or the tooling, and it is often more informative than the tasks that pass consistently. LLM determinism and seeds covers how far reproducibility goes.

Compare against a baseline, always

An absolute pass rate is nearly meaningless in isolation. Seventy per cent is excellent for a hard task and unacceptable for a trivial one.

Keep a stored baseline result and report every run as a delta against it. That converts an unanchored number into a decision: this change improved the rate on three tasks, degraded it on one, left the rest inside the noise.

Re-baseline deliberately and record when you did, along with the model version in use. A suite whose baseline drifts silently produces comparisons that look valid and are not. Pinning model versions covers the failure that causes.

Start with three tasks

A harness with three real tasks, run five times each, comparing against a baseline, is more useful than a plan for a comprehensive suite that never ships.

Pick tasks that represent work you actually run and that have unambiguous outcome checks. Add a task each time a production failure surprises you, using the failing case as the fixture. The suite then grows along the dimensions where your system is genuinely weak. Evaluating agent reliability covers what to do with the resulting numbers.