Terminal-Bench Leaders: The Benchmark That Predicts Agents

Most coding benchmarks hand the model a problem and grade the answer. Terminal-Bench hands it a shell and grades what happens next. That difference makes it the single best public predictor of how a model behaves inside an agent.

It is also the benchmark whose ordering most often diverges from the SWE-bench ordering, which is exactly why it is worth reading separately rather than as a confirmation.

What the benchmark actually does

The model is placed in a containerised environment with a real terminal and given a task described the way a colleague would describe it. It must work out what to run, run it, read the output, and decide what to do next. Success is measured by the state of the environment at the end, not by the text the model produced.

That construction exercises a specific stack of skills. The model has to translate an intent into a command, form the tool call correctly, interpret output that may be an error, a warning or silence, and revise its plan when the environment disagrees with its assumptions.

Crucially, nothing is well-specified. There is no test suite defining correctness in advance and no issue description enumerating the symptoms. That is much closer to the conditions under which you will actually run an agent.

The 2.1 leaders

On Terminal-Bench 2.1, Kimi K3 reports 88.3. GLM-5.2 reports 81.0. Those are the two figures to anchor on in the current open-weight field.

Kimi K3 is a mixture-of-experts model with roughly 2.8T total and 104B active parameters, which is the highest active count in the open-weight field and the most plausible mechanical explanation for the lead. Long tool-driven runs reward per-token reasoning depth more than most workloads do, because each decision constrains every decision after it.

GLM-5.2 gets within about seven points at roughly 40B active and a substantially lower price. That ratio — most of the capability for a fraction of the token cost — is the argument for routing rather than standardising on the top of the table. The GLM-5.2 guide covers where it holds up.

Why the ordering differs from SWE-bench

A model can be excellent at producing a correct patch when handed a precise issue and mediocre at deciding what to do when a command returns an unexpected error. Those are separable skills, and the two benchmarks separate them.

SWE-bench rewards code comprehension and patch correctness. Terminal-Bench rewards planning, tool-call formatting and recovery. A model trained heavily on repository data may lead one and trail the other.

The practical consequence is that you should read both and weight them according to how you deploy. If you are generating patches from well-written tickets, weight SWE-bench. If you are running an autonomous loop with shell access, weight Terminal-Bench heavily. The SWE-bench leaders are a different ranking on purpose.

What a high score does not guarantee

Terminal-Bench scores are produced under the benchmark's own harness, with its own tool definitions and its own prompt. Your harness is different, and the difference is not cosmetic.

Tool-call reliability is sensitive to schema design in ways that are easy to underestimate. A model that formats calls perfectly against clean, well-named, well-documented schemas can degrade noticeably against sprawling ones with overlapping parameters and vague descriptions. The benchmark measures the model against good schemas; your agent may not offer it any. Designing tool schemas covers what "good" means concretely here.

The benchmark also runs in a sandbox with a clean environment and no consequences. Your agent may be operating on a real repository where a destructive command is genuinely destructive, which changes the guard rails you need regardless of the score. Sandboxing an agent covers the containment side.

Run the failure test yourself

The cheapest useful supplement to the published number is a deliberate failure test on your own setup, and it takes about fifteen minutes.

Give the model a task that requires a tool which you have arranged to fail — a missing binary, a permissions error, a path that does not exist. Then read the next three turns closely. You are looking for whether it reads the error, forms a specific hypothesis, and tries a genuinely different approach.

The failure modes worth catching are retrying the identical command with cosmetic changes, and silently proceeding as though the failed step succeeded. The second is much more expensive, because every subsequent turn is billed and built on a state that does not exist. Error recovery patterns covers how to structure the harness so the model has enough information to recover.

Reading the table sensibly

Check the version. Terminal-Bench has iterated, and scores from different versions are not comparable — a 2.1 figure and an earlier figure describe different task sets.

Check whether the score is independent or vendor-reported, and weight accordingly. Then treat gaps of a few points as a tie and decide on cost, latency and licence, which are the differences that will still be true next month.

The decision rule for agent work: shortlist on Terminal-Bench, verify tool-call formatting against your own schemas, run the failure test, and only then compare price. A model that scores two points lower and recovers cleanly from errors will finish more runs than one that scores higher and stalls. The broader agentic comparison puts both benchmarks alongside price.