Public
goharbor / harbor
Benchmark updated: 8/13/202630 Tasks
An open source trusted cloud native registry project that stores, signs, and scans content.
Languages
Go53.2%TypeScript21.4%HTML7.8%Python5.4%RobotFramework5%Jinja3.8%Other3.3%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 18 / 30 | $3.04 | $5/$25 | 10m14s | ||
2 | Mini-SWE-agent | 17 / 30 | $3.59 | $2/$10 | 14m40s | ||
3 | Mini-SWE-agent | 16 / 30 | $0.41 | $2/$6 | 4m04s | ||
4 | Mini-SWE-agent | 14 / 30 | $0.33 | $1/$5 | 3m11s |
Key Takeaways
- Claude Sonnet 5 with Mini-SWE-agent resolves 17 tasks, followed by Grok 4.5 with Mini-SWE-agent at 16 and Claude Haiku 4.5 (Nonthinking) with Mini-SWE-agent at 14.
- Grok 4.5 with Mini-SWE-agent has the lowest cost per test at $0.41 and lowest latency at 244.04 seconds among the supplied runs.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
Claude Opus 5
Claude Haiku 4.5 (Nonthinking)
Grok 4.5
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 60.00% for $3.04 per test. No other model in this comparison is cheaper.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 5
Claude Opus 5
Grok 4.5
Claude Haiku 4.5 (Nonthinking)
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 60.00%, while Claude Haiku 4.5 (Nonthinking) is fastest at 3m 11s with 46.67% accuracy.