Public
harbor-framework / terminal-bench
Benchmark updated: 8/13/202630 Tasks
Measuring and evolving with the frontier of agent work
Languages
Python48.8%JavaScript14%Lean12%TypeScript9.3%Shell4.4%Dockerfile2.2%Other9.3%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 10 / 30 | $0.22 | $1/$5 | 2m46s |
Key Takeaways
- This single 30-task run provides no comparison; its result is directional rather than statistically significant.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Haiku 4.5 (Nonthinking)
Cost is the clearest tradeoff in this comparison. Claude Haiku 4.5 (Nonthinking) leads at 33.33% for $0.22 per test. No other model in this comparison is cheaper.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Haiku 4.5 (Nonthinking)
Claude Haiku 4.5 (Nonthinking) is both the most accurate and fastest model in this comparison at 33.33% and 2m 46s.