Background
Terminal-Bench-Science is an open-source
benchmark of 70 research-workflow tasks contributed by practicing scientists. Each task asks a model to
carry out a piece of real research work inside a sandboxed terminal — reconstructing a signal, calibrating
a model, reducing an instrument’s output, proving a bound — and is graded by verifier tests written by the
task’s own author. We score all 70 tasks from upstream v0.1.0.
Where Terminal-Bench 2.1 measures general terminal and software-engineering skill, this benchmark asks whether a model can do the work of a domain researcher. People who do that work wrote the tasks, and a verifier grades the artifact on whether it is scientifically correct, not on whether the model looked like it was trying.
Tasks are grouped by scientific domain rather than by difficulty tier:
| Domain | Tasks |
|---|---|
| Life Sciences | 19 |
| Physical Sciences | 17 |
| Mathematical Sciences | 17 |
| Engineering Sciences | 9 |
| Earth Sciences | 8 |
Methodology
This board is moving from the Terminus 2 agent to mini-swe-agent, the harness we use for Terminal-Bench 4.0. Most models have already been re-run on mini-swe-agent; the rest still show their Terminus 2 results until they are. The Harness column shows which one produced each row, and the harness filter limits the table to one. Terminus 2 has the model send keystrokes to a tmux session and read back the screen; mini-swe-agent gives it a single bash tool and returns each command’s output. The tasks themselves run through Harbor, the framework this dataset targets: it builds each task’s own container environment and runs the task’s verifier suite in a sandbox separate from the one the agent worked in, so a model cannot reach its own grader.
All results are pass@1 from a single run per model. Where a task ended in an infrastructure or provider error rather than a graded result, we re-attempted it; we never re-ran a task that produced a graded result. A handful of tasks were solved on such a re-attempt, so a strict one-attempt-per-task reading would place some rows slightly lower.
Our numbers are therefore not directly comparable to the official 0.1 leaderboard, which pairs each model with its own native agent — Claude Code, Codex, Grok Build — and averages three independent trials per task. We use one general-purpose harness instead of each vendor’s own, so a difference between two rows on the same harness reflects the model rather than the scaffold it shipped with. Expect our figures to sit below the official ones for the same model in most cases, since a vendor’s own agent is tuned for it.
Scoring
Each task’s own verifier suite scores it strictly pass/fail, and the reported figure is the percentage of the 70 tasks passed. No task earns partial credit. A task that produces a nearly correct result scores zero, which is deliberate: a nearly right research result is unusable.
Cost and runtime are reported for the benchmark as a whole rather than per domain. A run reports one total for each, and dividing either by a single domain’s task count would overstate it by the ratio between that domain and the full set.
Notes on difficulty
These tasks are markedly harder and slower than a conventional terminal benchmark, and the setup reflects that. Each task allows an eight-hour agent phase, against 15–60 minutes on Terminal-Bench 2.1, and tasks may request up to 4 vCPU and 16 GB. Several run multi-container environments and are graded on artifacts produced by a service other than the one the agent works in.
Expect low absolute scores. The benchmark is calibrated against what the contributing scientists consider a correct result rather than so that a good model passes most tasks. On that standard, current models solve a small fraction of the set.