Terminal-Bench Science

Key Takeaways

• GPT-6 Astra leads at 65.71%, nearly twice the next model, Claude Fable 5.1 (34.29%). Below those two the field compresses hard: twelve of twenty-four models sit at or under 4.29%, and three score zero.
• Astra tops four of the five domains and ties Fable 5.1 on engineering sciences (33.33% each), the hardest domain here at a 6.02% mean across the field.
• Cost per task spans a factor of more than ninety, from $0.56 to $51.62, and does not track score: GLM 5.3 spends $24.91 a task to reach 4.29%, while GPT-5.6 Luna scores higher at 5.71% for $0.56. These tasks allow an eight-hour agent phase, so a model that keeps working is charged for it whether it converges.

Background

Terminal-Bench Science is an open-source benchmark of 70 research-workflow tasks contributed by practicing scientists. Each task asks a model to carry out a piece of real research work inside a sandboxed terminal — reconstructing a signal, calibrating a model, reducing an instrument’s output, proving a bound — and is graded by verifier tests written by the task’s own author. We score all 70 tasks from upstream v0.1.0.

Where Terminal-Bench 2.1 measures general terminal and software-engineering skill, this benchmark asks whether a model can do the work of a domain researcher. People who do that work wrote the tasks, and a verifier grades the artifact on whether it is scientifically correct, not on whether the model looked like it was trying.

Tasks are grouped by scientific domain rather than by difficulty tier:

Methodology

We benchmarked every model with the Terminus 2 agent, in the same configuration we use for Terminal-Bench 2.1, so the two boards differ in their tasks rather than in how the model is driven. The tasks themselves run through Harbor, the framework this dataset targets: it builds each task’s own container environment and runs the task’s verifier suite in a sandbox separate from the one the agent worked in, so a model cannot reach its own grader.

All results are pass@1 from a single run per model. Where a task ended in an infrastructure or provider error rather than a graded result, we re-attempted it; we never re-ran a task that produced a graded result. A handful of tasks were solved on such a re-attempt, so a strict one-attempt-per-task reading would place some rows slightly lower.

Our numbers are therefore not directly comparable to the official 0.1 leaderboard, which pairs each model with its own native agent — Claude Code, Codex, Grok Build — and averages three independent trials per task. We hold the harness fixed instead, so a difference between two rows here reflects the model rather than the scaffold it shipped with. Expect our figures to sit below the official ones for the same model in most cases, since a vendor’s own agent is tuned for it.

Scoring

Each task’s own verifier suite scores it strictly pass/fail, and the reported figure is the percentage of the 70 tasks passed. No task earns partial credit. A task that produces a nearly correct result scores zero, which is deliberate: a nearly right research result is unusable.

Cost and runtime are reported for the benchmark as a whole rather than per domain. A run reports one total for each, and dividing either by a single domain’s task count would overstate it by the ratio between that domain and the full set.

Notes on difficulty

These tasks are markedly harder and slower than a conventional terminal benchmark, and the setup reflects that. Each task allows an eight-hour agent phase, against 15–60 minutes on Terminal-Bench 2.1, and tasks may request up to 4 vCPU and 16 GB. Several run multi-container environments and are graded on artifacts produced by a service other than the one the agent works in.

Expect low absolute scores. The benchmark is calibrated against what the contributing scientists consider a correct result rather than so that a good model passes most tasks. On that standard, current models solve a small fraction of the set.