Key Takeaways
- Claude Opus 5.5 (65.15%) and Claude Sonnet 5.5 (64.14%) sit a point apart at the top, ahead of GPT-6 Astra (59.60%) and Claude Fable 5.1 (58.08%). Even the leader fails about a third of the tasks, and only eleven of the thirty models clear 30%.
- Category strengths differ even among the leaders. Opus 5.5 is strongest on science (71.43%, fourteen points clear of Fable 5.1) and ties Sonnet 5.5 on ML (81.82%), but drops to 29.63% on operations, where Sonnet 5.5 leads at 51.85%; Astra is the only model above 60% on software (61.11%). Security is the most solvable category at the top — Claude Opus 5 and Opus 5.5 both score 100% — while hardware is the hardest across the field: sixteen of thirty models score zero on it, and the field mean is 15.1%.
- Cost per task tracks score only loosely. Astra places third at $9.58 a task, cheaper than both Opus 5.5 ($13.20) and Sonnet 5.5 ($16.51), while Claude Fable 5 is the most expensive run on the board at $30.34 for 41.41%. At the low end, MiMo V2.6 Pro reaches 31.31% at $0.50 a task, though it is the second-slowest run on the board. With an eight-hour agent phase, a model that keeps working without converging is charged for it.
- Five Anthropic rows include provider-side fallback. Opus 5.5 had 22 of 198 task attempts served by Opus 5 or Claude Opus 4.8; counting those as failures lowers it from 65.15% to 58.08%, behind Sonnet 5.5 and Astra. Counted the same way, Fable 5.1 (22 attempts served by Opus 5) drops to 50.00%, Fable 5 (42 by Opus 4.8) to 33.84%, Sonnet 5.5 (4 by Claude Sonnet 5) to 62.63%, and Opus 5 (1 by Opus 4.8) to 53.03%. Where no fallback served the task, provider refusals — nine on Qwen 3.8 Max and three each on Sonnet 5 and Gemini 3.8 Flash — are scored as failures. Both are itemized in the refusal and fallback icons on each affected row of the table.
Background
Terminal-Bench is an open-source benchmark that measures how well an agent can complete long, realistic pieces of work in a sandboxed terminal. Version 4.0 is a fresh set of 66 community-contributed, maintainer-reviewed tasks, none of which appear in Terminal-Bench 2.1. Each task is built around a real deliverable — a working service, a proof, a CAD model, a trained kernel, a forensic report — and carries an expert time estimate: the median task is estimated at 4 hours of expert work, and the range runs from under an hour to 60 hours.
Where Terminal-Bench 2.x was mostly software engineering and system administration, 4.0 is deliberately wide. The tasks span seven categories:
- Software (18 tasks): databases, distributed systems, frontend performance, data engineering, algorithms
- Science (14 tasks): biology, chemistry, physics, formal mathematics, robotics, linguistics, earth science
- ML (11 tasks): training runs, inference serving, GPU kernels, evaluation harnesses
- Operations (9 tasks): logistics, finance, insurance claims, compliance, supply chain
- Hardware (5 tasks): CAD modeling and RTL design
- Security (5 tasks): cryptography, application security, forensics
- Media (4 tasks): music theory, audio transcription, layout design
We include Terminal-Bench because it is widely reported by model providers, because it reflects the kind of end-to-end terminal work that agentic coding products (Claude Code, Codex, Cursor and others) are expected to do, and because 4.0 raises the ceiling considerably: the tasks are long, multi-step, and graded strictly on the final artifact.
Terminal-Bench is developed by the Terminal-Bench community and the Harbor team as an open-source effort. We thank the community for building the benchmark and for helping us integrate it into our evaluation suite. To learn more or contribute tasks, visit tbench.ai.
Below is an example task from the Operations category (you can find the full task on the Harbor Hub).
Speed up the worker startup for the financial transaction processing pipeline so overdraft notifications are correct, not missed or duplicated, and arrive within 5s during worker respawns and rolling deploys in fresh containers.
Add any extra Python packages to
/app/src/worker/requirements.txt.
Methodology
All models were benchmarked using mini-swe-agent, a minimal, model-agnostic agent harness. The model is given the task instruction and a single tool, bash: each turn it issues one or more shell commands, which run in a fresh subshell inside the task sandbox, and it reads back their output. There is no step or cost limit; the agent works until it signals that it has finished or reaches the task’s time limit.
All results reported are avg@3: each model is run through the full benchmark three times, each run scored as pass@1 (one attempt per task), and the displayed score is the mean of the three run-level scores, with error bars showing the standard error across those three runs. Each task is graded by its own verifier, which runs after the agent finishes and inspects the final state of the environment or the artifact the agent produced. A model must pass the verifier’s full test suite to get any credit for a task — there is no partial credit.
Tasks are run in isolated sandboxes on Daytona, following the upstream Harbor setup, including the three tasks that require a GPU. Each task has an 8-hour agent time limit, and each verifier has its own task-specific timeout; an agent that exceeds the time limit, or a verifier that cannot produce a result from the agent’s submission, scores zero for that task, matching Harbor’s semantics.
Comparison to Terminal-Bench 2.1
Our Terminal-Bench 2.1 results used the Terminus 2 harness; Terminal-Bench 4.0 uses mini-swe-agent. The Daytona sandboxes and pass@1 scoring are the same. The task set also changed:
- New tasks. All 66 tasks are new; there is no overlap with the 89 tasks in 2.1.
- Categories instead of difficulty tiers. 2.1 tagged tasks as easy/medium/hard. 4.0 tags them by domain (the seven categories above) and by an expert time estimate. We report scores per category.
- Long time limits. Every 4.0 task allows the agent 8 hours.
- Broader scope. 2.1 was predominantly software engineering. Roughly three quarters of 4.0 is outside traditional software engineering — science, ML systems, operations, hardware, security, and media.
Because the task set is entirely different, scores are not comparable across versions.