Public
NVIDIA / nccl
Benchmark updated: 8/13/202630 Tasks
Optimized primitives for collective multi-GPU communication
Languages
C++53.6%Cuda18%C13.6%Python9%Cython2.3%Rust1.6%Other1.9%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 14 / 30 | $2.96 | $5/$25 | 9m56s | ||
2 | Mini-SWE-agent | 13 / 30 | $2.35 | $4/$20 | 9m13s | ||
3 | Mini-SWE-agent | 11 / 30 | $0.67 | $1.25/$4.25 | 6m07s |
Key Takeaways
- GPT-5.6 Sol with Mini-SWE-agent resolves 13 of 30 tasks, one behind Claude Opus 5, at $2.35 per test and 552.64 seconds.
- Muse Spark 1.2 with Mini-SWE-agent resolves 11 of 30 tasks, with the lowest cost at $0.67 per test and latency of 367.22 seconds.
- These 30-task results are directional comparisons across the supplied Mini-SWE-agent runs.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Muse Spark 1.2
Claude Opus 5
GPT-5.6 Sol
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 46.67% for $2.96 per test. GPT-5.6 Sol is the lower-cost option at 43.33% for $2.35 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Opus 5
GPT-5.6 Sol
Muse Spark 1.2
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 46.67%, while Muse Spark 1.2 is fastest at 6m 7s with 36.67% accuracy.