Building a Model Eval Harness You Will Actually Maintain

Most teams run a model comparison once, in a spreadsheet, and never repeat it. Three months later a new model ships, nobody remembers the prompt, and the whole thing gets redone from scratch or skipped.

A harness is the fix, and it is much smaller than people assume. A few hundred lines gets you something you can point at any candidate and get a comparable answer from in twenty minutes.

The fixture format

Keep each task as a directory rather than a row in a file. One directory holds the prompt, any input files, the expected outcome, and the script that decides pass or fail. Directories are diffable, reviewable in a pull request, and survive the format changes that kill spreadsheets.

The prompt file should contain only the task statement, exactly as a developer would have received it — the issue text, the incident description, the change request. Anything you add to help is scaffolding and belongs in the runner, applied identically to every candidate.

Store the ground truth alongside it. For a code task that is the real diff; for a document task it is the deliverable someone actually produced. You will not always score against it mechanically, but you will constantly want to look at it.

Add a small metadata file: task type, rough difficulty, which part of the system it touches, and when it was added. That metadata is what lets you say "we regressed on database tasks" instead of "the score went down".

Isolation and setup

Every run needs a clean starting state, and the cheapest way to get one is a container per task with the repository checked out at the parent commit. Without isolation you will spend most of your debugging time working out whether a failure was the model or leftover state.

Pin the environment as tightly as you pin the model. Dependency versions, tool versions, and the base image all affect outcomes, and a harness whose environment drifts produces score changes you will misattribute to the model.

Give the model the same tool surface every time. If it gets a shell, define exactly which commands are available and what the turn limit is, because those limits are part of the measurement — the same reason Terminal-Bench figures are harness-dependent.

Sandbox properly rather than trusting the model to behave. It will occasionally run something destructive, and you want that to cost a container rather than a workstation. Agent sandboxing covers the boundaries worth enforcing.

Scoring

Prefer a mechanical check wherever one exists, because it is reproducible and cannot be argued with. For code, run the tests that the real fix made pass and record the result — this is the same methodology as SWE-bench, applied to your distribution.

Where no mechanical check exists, use a rubric with a small number of concrete criteria rather than a single quality rating. Three binary questions produce far more consistent results across graders than one score out of ten.

If you use a model as a grader, treat it as an instrument that needs calibrating. Grade twenty items by hand, compare, and measure agreement before you trust it on the rest. A judge that disagrees with your reviewers a third of the time is generating noise with a confident interface.

Always write two scores per task, not one: did it pass, and would you have merged it. The gap between those columns is the most informative output the harness produces.

Recording runs

Persist the full transcript for every attempt — prompt, every tool call, every response, the final artefact. When a result surprises you, the transcript is the only thing that explains it, and reconstructing one is impossible.

Record the cost dimensions alongside: input tokens, output tokens, cached input tokens, turn count and wall-clock time. Store tokens rather than currency so the record stays valid when prices move, and note that cached input can be several times cheaper — Kimi K3, for instance, prices cached input at $0.30 against $3 for uncached, which changes the arithmetic on repeated context.

Stamp every run with the exact model identifier you called, not the family name. "The Kimi one" is not a record, and pinning model versions is what makes two runs comparable at all.

Keep results in a plain file format in the repository. A database is nicer until the person who set it up leaves; a directory of JSON files survives everything.

Wiring it into the workflow

Do not run the full set on every commit — it costs real money and most commits do not touch model behaviour. Run it on a schedule, and on any change to prompts, tool schemas, context assembly or model configuration.

Make the output a diff against the last stored run rather than a fresh score. What you care about is which tasks changed status, and a list of three newly failing tasks is actionable in a way that a two-point aggregate drop is not.

Set a budget ceiling in the runner itself and fail loudly when a task blows through it, because a model that loops will happily spend your evaluation budget on one fixture. Agent token budgets covers the enforcement patterns.

Then treat the harness as production code: reviewed, tested, owned by someone. A harness nobody owns produces numbers nobody trusts, and the whole point was to have an answer you can act on when the next version ships.