Terminal-Bench Explained: Scoring Agents That Run Commands

Terminal-Bench measures something SWE-bench does not: whether a model can be left alone in a shell for a long stretch and still arrive somewhere useful. That is a different capability from producing a good patch, and it ranks models differently.

It has quietly become the figure people reach for when they mean "is this thing actually usable as an agent". That makes it worth understanding properly, including the parts of agent behaviour it still cannot see.

What the benchmark actually runs

Each task is a container with a defined starting state and an objective written in plain language. The model is handed a terminal. It issues commands, reads whatever comes back, and decides what to do next, turn after turn, until it declares itself finished or exhausts its budget.

Then a scoring script runs inside that container and checks whether the objective was met. Not whether the transcript looked competent, not whether the reasoning was sound — whether the file exists, the service answers, the test passes.

The tasks range well beyond editing code. Installing and configuring software, repairing a deliberately broken environment, wrangling data across several formats, tracking down why a process refuses to start, building something small from a thin description.

Version 2.1 is the revision most current figures cite. Treat a bare "Terminal-Bench" number with the same suspicion you would give a bare SWE-bench number, since the variant decides what the figure means.

Why it separates models that look similar elsewhere

The mechanism is compounding. A bounded benchmark asks for one output, so a mistake costs you one task. A shell task with thirty steps feeds each step's output into the next step's input, so a per-step error rate multiplies rather than averages out.

Two models a couple of points apart on single-turn accuracy can be far apart here, because the one that recovers from a bad command instead of building on it finishes runs the other abandons. That recovery behaviour is most of what the score is measuring.

It also punishes a failure mode benchmarks rarely catch: confident drift. A model that misreads an error message and then spends fifteen turns fixing the wrong thing scores zero, exactly like a model that gave up immediately, even though the transcript looks industrious.

If your workload is long unattended runs rather than single well-specified edits, this is the more predictive number of the two. Why benchmarks disagree covers how horizon length drives most ranking conflicts.

How to read a score

Read it as a completion rate under a specific harness, not as a general competence rating. The harness sets the turn limit, the timeout, the tool surface and the system prompt, and all four move the number.

Among the figures currently published, Kimi K3 reports 88.3 on Terminal-Bench 2.1 and GLM-5.2 reports 81.0. Those are close enough that harness differences plausibly account for part of the gap, and far enough apart to be worth a look if you run long agent sessions.

What a score does not encode is cost. A model can reach the same completion rate by taking three times as many turns, and turns are tokens. When you compare, ask for tokens per completed task alongside the percentage — the cost of running an agent fleet is decided by that ratio, not by the headline.

It also says nothing about how the model behaves when it fails. An agent that stops and reports uncertainty is operationally very different from one that keeps issuing destructive commands, and both register as a failed task.

What it does not measure

Safety of the actions taken. The container is disposable, so nothing punishes a model for running a broad delete or rewriting files it was not asked to touch, provided the objective still ends up satisfied. In your repository that behaviour is expensive.

Interaction with a human. Every task is fully unattended by design, so a model that would have asked one clarifying question and saved twenty turns gets no credit for that instinct. Most real agent deployments are not unattended, and human-in-the-loop design changes which model wins.

Your stack. The environments are Linux containers with common tooling. If your work happens inside a large private monorepo with bespoke build scripts, the transfer is partial at best.

Using it in a real decision

Use it as a shortlist filter and a profile indicator, in that order. A model well behind the field here is unlikely to hold together across a forty-step run on your code, so it is a fair first cut.

Then read the shape rather than the ranking. A model that leads on bounded coding benchmarks but trails here has depth without consistency: good for one-shot edits, risky as an autonomous agent. The reverse profile is the one you want supervising a long refactor.

Finally, replace it. Take ten genuine operational tasks from your own history — a failed migration, an environment that would not build, a flaky test — write a checking script for each, and run your candidates in a container. Building a model eval harness covers the mechanics, and that private set is the only leaderboard whose ordering you should act on.