Public
JudgmentLabs / judgeval
Benchmark updated: 9/16/202630 Tasks
The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
Languages
Python100%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 28 / 30 | $4.13 | $2/$10 | 14m39s |
Key Takeaways
- This 30-task result is a single-run benchmark summary with no supplied comparison.
- Claude Sonnet 5 with Mini-SWE-agent leaves 2 of 30 tasks unresolved.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
Cost is the clearest tradeoff in this comparison. Claude Sonnet 5 leads at 93.33% for $4.13 per test. No other model in this comparison is cheaper.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 5
Claude Sonnet 5 is both the most accurate and fastest model in this comparison at 93.33% and 14m 39s.