Key Takeaways
- Claude Opus 5.5 leads at 87.64%, a fraction ahead of GPT-6 Astra (87.27%), then GPT-5.6 Sol (85.77%), Claude Fable 5.1 (85.02%) and Claude Opus 5 (84.64%). The two leaders get there differently: Opus 5.5 takes the medium split at 90.30% but sits fifth on hard at 81.11%, while Astra’s edge is the hard split, where it reaches 87.78% against 84.44% for Sol and Opus 5 and 82.22% for Fable 5.1, with a sharp cliff to sixth place (74.44%); on medium Astra only ties Sol and Claude Sonnet 5.5 at 86.06%. While seventeen models reach 100% on easy, the median score on that split is 83.33%, so even the easy tasks still separate the field.
- Note on the headline scores: Opus 5.5’s run used Opus 5 and Claude Opus 4.8 as fallbacks on 26 of 267 tasks; counting those as failures lowers it from 87.64% to 79.77%, behind Astra. Both Fable 5’s and Opus 5’s runs also used Opus 4.8 as a refusal fallback; counting Opus 5’s nine affected passes as failures lowers it from 84.64% to 81.27% (toggle the split icon beside the score to compare).
Background
Terminal-Bench 2.1 is an open-source benchmark that is designed to test a model’s ability to navigate and complete tasks in a sandboxed terminal environment. This version of the benchmark uses 89 tasks with unique categories ranging from model training to system administration. These tasks scale in difficulty from easy to hard.
We chose to include Terminal-Bench because a) it is increasingly common for it to be reported by model providers, b) it reflects the real-world terminal tasks expected of software engineers, and c) it is quite challenging, with no model scoring above 50% on the hard tasks upon its initial release. Furthermore, agentic systems like Claude Code, Codex, and Cursor now rely heavily on executing terminal commands correctly.
This benchmark was developed by the Terminal-Bench community as an open-source effort; we’d like to thank the community for their efforts in building this benchmark and for helping us integrate it into our evaluation suite. If you’re interested in learning more about Terminal-Bench or want to contribute to the project, visit tbench.ai
Below is an example task (you can find the full details for this task in the open-source task registry).
Please train a fasttext model on the yelp data in the data/ folder.
The final model size needs to be less than 150MB but get at least 0.62 accuracy on a private test set that comes from the same yelp review distribution.
The model should be saved as /app/model.bin
Methodology
All models were benchmarked using the Terminus 2 harness. Unless otherwise specified, we use identical configuration and methodology to Terminus 2. All results reported are pass@1.
On submission, we run the model against the provided pytests—a model must pass all pytests to get any credit for a task.
Unlike Terminus 1, Terminus 2 does not use structured outputs to enforce a response schema. Model queries returning an invalid or missing JSON are retried with a warning.
Comparison to Original Terminal-Bench
This benchmark is similar in structure to the original Terminal-Bench - it features 80 terminal-based tasks, many of which (like our example!) also appear in Terminal-Bench 2.1. The most substantial differences are in evaluation methododology:
- We ran the original Terminal-Bench on an EC2 instance, using docker containers. Following the Laude implementation, we run Terminal-Bench 2.1 remotely using daytona.
- We ran the original Terminal-Bench using a turn limit. Again, following the Laude implementation, we run Terminal-Bench 2.1 using a time limit instead.