Public
chrisanderton / tyk-gateway
Benchmark updated: 8/16/202630 Tasks
Tyk Open Source API Gateway written in Go, supporting REST, GraphQL, TCP and gRPC protocols
Languages
Go98%Shell0.7%Python0.6%JavaScript0.3%Java0.2%Dockerfile<0.1%Other0.3%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 23 / 30 | $1.42 | $4/$20 | 5m38s | ||
2 | Mini-SWE-agent | 23 / 30 | $3.00 | $5/$25 | 10m40s | ||
3 | Mini-SWE-agent | 22 / 30 | $3.21 | $5/$25 | 10m45s | ||
4 | Mini-SWE-agent | 20 / 30 | $0.49 | $1.4/$4.4 | 7m26s |
Key Takeaways
- GPT-5.6 Sol matches Claude Opus 5 while reporting lower cost per test ($1.42 versus $3.00) and lower latency (337.56 versus 639.94 seconds) with Mini-SWE-agent.
- Claude Opus 4.8 resolves 22 of 30 tasks for 73.33% with Mini-SWE-agent, while GLM 5.2 (Fireworks) resolves 20 for 66.67%; these 30-task results are directional.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Opus 4.8
GPT-5.6 Sol
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 76.67% for $3.00 per test. GPT-5.6 Sol is the lower-cost option at 76.67% for $1.42 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Opus 4.8
Claude Opus 5
GLM 5.2
GPT-5.6 Sol
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 76.67%, while GPT-5.6 Sol is fastest at 5m 38s with 76.67% accuracy.