Public
mkarots / benedict
Benchmark updated: 10/1/202630 Tasks
Bene(volent) dict(ator) agents for repos
Languages
Python91.6%HTML7.5%Makefile0.9%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 24 / 30 | $1.45 | $3/$15 | 8m22s |
Key Takeaways
- This single 30-task run provides no model or harness comparison.
- Claude Sonnet 4.6 with Mini-SWE-agent resolves 80% of the supplied tasks.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
No token usage data available.
Cost is the clearest tradeoff in this comparison. Claude Sonnet 4.6 leads at 80.00% for $1.45 per test. No other model in this comparison is cheaper.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 4.6
Claude Sonnet 4.6 is both the most accurate and fastest model in this comparison at 80.00% and 8m 22s.