Public
poalrom / sharepaste
Benchmark updated: 8/14/202630 Tasks
Custom repository benchmark.
Languages
Kotlin40.5%Rust34.8%TypeScript23.6%CSS0.7%Makefile0.2%JavaScript0.1%Other<0.1%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 15 / 30 | $3.53 | $5/$25 | 11m10s | ||
2 | Mini-SWE-agent | 11 / 30 | $1.97 | $4/$20 | 7m47s |
Key Takeaways
- Claude Opus 5 with Mini-SWE-agent scores 50% versus GPT-5.6 Sol with Mini-SWE-agent at 36.67%.
- GPT-5.6 Sol with Mini-SWE-agent costs $1.97 per test and takes 466.55 seconds, compared with $3.53 and 670.47 seconds for Claude Opus 5.
- With 30 tasks, these are directional benchmark results rather than a smoke test.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Opus 5
GPT-5.6 Sol
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 50.00% for $3.53 per test. GPT-5.6 Sol is the lower-cost option at 36.67% for $1.97 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Opus 5
GPT-5.6 Sol
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 50.00%, while GPT-5.6 Sol is fastest at 7m 47s with 36.67% accuracy.