Public
Expertium / fsrs-autoresearch
Benchmark updated: 9/22/202630 Tasks
Improving FSRS-7 with Claude
Languages
Python92.4%Cuda6.2%C++1.3%Shell<0.1%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 29 / 30 | $0.63 | N/A | 2m10s | ||
2 | Mini-SWE-agent | 29 / 30 | $0.57 | $2/$10 | 2m23s | ||
3 | Mini-SWE-agent | 28 / 30 | $0.46 | N/A | 101.25s | ||
4 | Mini-SWE-agent | 25 / 30 | $0.24 | $1/$5 | 2m29s |
Key Takeaways
- Claude Opus 5 (High Effort) with Mini-SWE-agent is faster at 130.12 seconds, while Claude Sonnet 5 (High Effort) costs less at $0.57 per test.
- Claude Opus 4.8 (High Effort) with Mini-SWE-agent resolves 28 of 30 tasks, followed by Claude Haiku 4.5 (Nonthinking) with 25 of 30.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Haiku 4.5 (Nonthinking)
Claude Sonnet 5
anthropic/claude-opus-5-high
anthropic/claude-opus-4-8-high
Cost is the clearest tradeoff in this comparison. anthropic/claude-opus-5-high leads at 96.67% for $0.63 per test. anthropic/claude-opus-4-8-high is the lower-cost option at 93.33% for $0.46 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Haiku 4.5 (Nonthinking)
Claude Sonnet 5
anthropic/claude-opus-5-high
anthropic/claude-opus-4-8-high
Latency separates several models with similarly strong scores. anthropic/claude-opus-5-high leads at 96.67%, while anthropic/claude-opus-4-8-high is fastest at 1m 41s with 93.33% accuracy.