Verifier Agents: Gating Completion on a Check That Passes

The default stopping rule in an agent loop is that the model stops calling tools. That treats the model's own judgement about completion as authoritative, and completion judgement is among the least reliable things a model produces.

A verifier replaces that rule. Before completion is accepted, something independent checks that the stated goal was met, and a failure sends the agent back to work with the reason. It is a small change to the loop and one of the largest reliability gains available.

Deterministic verifiers come first

A verifier does not have to be a model, and where it can be code it should be.

Did the tests pass. Does the project compile. Does the output validate against the schema. Does the endpoint return the expected status. Each is fast, repeatable, cannot be talked out of its answer, and costs nothing per invocation.

Most tasks have more deterministic verification available than teams use, because the checks exist in CI and were never wired into the agent loop. Moving them earlier so the agent runs them itself is usually the highest-return change in the whole system. The agent then finds its own mistakes in seconds rather than in a pipeline run twenty minutes later.

Model verifiers for the rest

Some goals cannot be checked by code: whether a refactor preserved behaviour, whether documentation matches the implementation, whether a change addresses the issue it claims to.

Here a model verifier is a separate call whose only job is to answer pass or fail against stated criteria. It sees the original task and the produced artefact, and it does not see the reasoning that produced them — that omission is the point, because inherited reasoning is what makes self-review weak.

Give it explicit criteria rather than asking whether the work is good. Does every public function added in this diff have a docstring is checkable; is this well documented is an invitation to agree. Agent self-correction covers why the framing matters.

Verifiers are not critics

The distinction is worth keeping sharp. A verifier returns a binary verdict against fixed criteria and gates the loop. A critic returns graded feedback intended to improve work that is already acceptable.

Conflating them produces a gate that never closes, because a critic can always find something to say and the loop keeps cycling on cosmetic feedback. Verifier criteria must be finite, enumerated in advance, and answerable without judgement about quality.

Run both if you want, but only the verifier blocks. Critic agent patterns covers the other half.

Cheaper models verify well

Verification is a narrower task than generation. The verifier is not solving the problem, only checking a stated property of a finished artefact against explicit criteria.

That asymmetry means a smaller, cheaper model is often adequate as a verifier even when a stronger one is needed to do the work. It also makes the economics comfortable: a verification pass on a fraction of the generation cost is easy to justify on every completion.

Use a different model from the generator where you can. Two runs of the same model share the same blind spots, and a verifier that fails in the same places as the generator is close to no verifier at all. When a cheap model is enough covers the sizing question.

The failure message is the whole interface

A verifier that returns fail has moved the agent no closer to done, and the retry will likely reproduce the same output.

Return which criterion failed, what was observed, and what was expected. That triple turns a rejection into a work item the model can act on directly, and it is the difference between a verifier that converges and one that just burns turns.

Cap the retry count. If verification fails three times on the same criterion, the agent is not going to satisfy it and the run should stop with the transcript and the verdict for a human. Agent timeout strategies covers bounding the cycle.

Verify the verifier

A verifier is a component with its own error rate, and an unmeasured one is a source of false confidence rather than assurance.

Build a small labelled set: artefacts known to pass and artefacts known to fail, including near-misses that are subtly wrong. Run the verifier over it and record both error types.

False passes are the dangerous half, because they let bad work through with a green light attached. False failures are merely expensive. Tune towards catching everything and accept some noise, since a false failure costs a retry and a false pass costs an incident.

Re-run the set when you change the verifier prompt or the model behind it. Verifier prompts drift under editing in ways that are invisible without a fixed set to check against. How to benchmark LLMs on your own work covers assembling one.

Where verification changes the economics

Verification is what makes unattended running defensible. Without it, an agent that runs overnight produces output nobody has checked and a human has to review all of it, which removes most of the benefit.

With a verifier gating completion, the overnight run either produces work that passed a check or stops with a reason. Reviewers then spend their attention on the exceptions rather than on everything. Evaluating agent reliability covers measuring the resulting pass rate.

The rule is straightforward: if you cannot state a check that would tell you the task succeeded, the task is not ready to be run unattended. Write the check first and the agent design follows from it.