Key Takeaways
- GPT-6 Astra leads at 57.07%, eight points clear of Claude Fable 5.1 (49.49%) and twelve clear of Claude Opus 5 (45.45%). Nobody else clears 30% of the tasks: the remaining twenty-four models range from 27.78% down to 0.00%, and eleven of them sit at or under 10%.
- Astra tops five of the seven categories, including 75.00% on media and 73.33% on hardware; Opus 5 leads ML at 63.64%, and Fable 5.1 and Opus 5 tie on security at 73.33% to Astra’s 53.33%. Hardware and media are the hardest categories for the field: fourteen of twenty-seven models score zero on each, and operations has the lowest mean at 11.5%.
- Cost per task does not buy score. GLM 5.3 Flash reaches 19.70% at $0.65 a task, ahead of Claude Sonnet 5 (8.08%, $21.37) and Claude Opus 4.8 (16.16%, $28.31), and Astra itself is one of the cheaper frontier runs at $8.21. With an eight-hour agent phase, a model that keeps working without converging is charged for it.
- Three Anthropic rows include provider-side fallback: across the three runs the gateway served part of Fable 5.1’s tasks with Opus 5 (28 of 198 task attempts; 42.42% on the requested model alone), part of Claude Fable 5‘s with Opus 4.8 (51 attempts; 16.67% alone), and a single Opus 5 attempt with Opus 4.8 (score unchanged). Provider refusals — most notably twenty-one on Qwen 3.8 Max and seven each on GPT-5.6 Sol and Claude Sonnet 5 — are scored as failures. Both are itemized in the refusal and fallback icons on each affected row of the table.
Background
Terminal-Bench is an open-source benchmark that measures how well an agent can complete long, realistic pieces of work in a sandboxed terminal. Version 4.0 is a fresh set of 66 community-contributed, maintainer-reviewed tasks, none of which appear in Terminal-Bench 2.1. Each task is built around a real deliverable — a working service, a proof, a CAD model, a trained kernel, a forensic report — and carries an expert time estimate: the median task is estimated at 4 hours of expert work, and the range runs from under an hour to 60 hours.
Where Terminal-Bench 2.x was mostly software engineering and system administration, 4.0 is deliberately wide. The tasks span seven categories:
- Software (18 tasks): databases, distributed systems, frontend performance, data engineering, algorithms
- Science (14 tasks): biology, chemistry, physics, formal mathematics, robotics, linguistics, earth science
- ML (11 tasks): training runs, inference serving, GPU kernels, evaluation harnesses
- Operations (9 tasks): logistics, finance, insurance claims, compliance, supply chain
- Hardware (5 tasks): CAD modeling and RTL design
- Security (5 tasks): cryptography, application security, forensics
- Media (4 tasks): music theory, audio transcription, layout design
We include Terminal-Bench because it is widely reported by model providers, because it reflects the kind of end-to-end terminal work that agentic coding products (Claude Code, Codex, Cursor and others) are expected to do, and because 4.0 raises the ceiling considerably: the tasks are long, multi-step, and graded strictly on the final artifact.
Terminal-Bench is developed by the Terminal-Bench community and the Harbor team as an open-source effort. We thank the community for building the benchmark and for helping us integrate it into our evaluation suite. To learn more or contribute tasks, visit tbench.ai.
Below is an example task from the Operations category (you can find the full task on the Harbor Hub).
Speed up the worker startup for the financial transaction processing pipeline so overdraft notifications are correct, not missed or duplicated, and arrive within 5s during worker respawns and rolling deploys in fresh containers.
Add any extra Python packages to
/app/src/worker/requirements.txt.
Methodology
All models were benchmarked using the Terminus 2 harness — the reference agent shipped with Harbor — using identical configuration and methodology to the upstream implementation unless otherwise specified. Terminus 2 is a minimal, model-agnostic agent: the model is given the task instruction and a tmux session, and each turn it emits keystrokes to run in the terminal and reads back the resulting screen. It does not use structured outputs; responses with invalid or missing JSON are retried with a warning.
All results reported are avg@3: each model is run through the full benchmark three times, each run scored as pass@1 (one attempt per task), and the displayed score is the mean of the three run-level scores, with error bars showing the standard error across those three runs. Each task is graded by its own verifier, which runs after the agent finishes and inspects the final state of the environment or the artifact the agent produced. A model must pass the verifier’s full test suite to get any credit for a task — there is no partial credit.
Tasks are run in isolated sandboxes on Daytona, following the upstream Harbor setup, including the three tasks that require a GPU. Each task has an 8-hour agent time limit, and each verifier has its own task-specific timeout; an agent that exceeds the time limit, or a verifier that cannot produce a result from the agent’s submission, scores zero for that task, matching Harbor’s semantics.
Comparison to Terminal-Bench 2.1
The evaluation methodology is unchanged from our Terminal-Bench 2.1 results — same Terminus 2 harness, same Daytona sandboxes, same pass@1 scoring. What changed is the task set:
- New tasks. All 66 tasks are new; there is no overlap with the 89 tasks in 2.1.
- Categories instead of difficulty tiers. 2.1 tagged tasks as easy/medium/hard. 4.0 tags them by domain (the seven categories above) and by an expert time estimate. We report scores per category.
- Long time limits. Every 4.0 task allows the agent 8 hours.
- Broader scope. 2.1 was predominantly software engineering. Roughly three quarters of 4.0 is outside traditional software engineering — science, ML systems, operations, hardware, security, and media.
Because the task set is entirely different, scores are not comparable across versions.