Public
Chen-Oliver / verifiers
Benchmark updated: 8/13/202630 Tasks
Our library for RL environments + evals
Languages
Python98.9%TypeScript0.7%Shell0.3%HTML<0.1%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 18 / 30 | $3.60 | $5/$25 | 12m50s |
Key Takeaways
- Claude Opus 5 with Mini-SWE-agent scores 60% across 30 tasks, making this more than a smoke test.
- The run reports 770.04 seconds of latency per task; token metrics were not supplied.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
No token usage data available.
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 60.00% for $3.60 per test. No other model in this comparison is cheaper.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Opus 5
Claude Opus 5 is both the most accurate and fastest model in this comparison at 60.00% and 12m 50s.