Public
vllm-project / llm-compressor
Benchmark updated: 9/24/202630 Tasks
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
Languages
Python99.4%Shell0.5%Makefile<0.1%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 29 / 30 | $1.93 | $5/$25 | 9m43s | ||
2 | Mini-SWE-agent | 28 / 30 | $0.69 | $4/$20 | 4m40s |
Key Takeaways
- GPT-5.6 Sol with Mini-SWE-agent resolves 28 of 30 tasks at $0.69 per test, versus Claude Opus 5 with Mini-SWE-agent at $1.93.
- Claude Opus 5 with Mini-SWE-agent records 583.42 seconds average latency, compared with 279.80 seconds for GPT-5.6 Sol with Mini-SWE-agent.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Opus 5
GPT-5.6 Sol
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 96.67% for $1.93 per test. GPT-5.6 Sol is the lower-cost option at 93.33% for $0.69 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Opus 5
GPT-5.6 Sol
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 96.67%, while GPT-5.6 Sol is fastest at 4m 40s with 93.33% accuracy.