Public
palantir / blueprint
Benchmark updated: 8/15/202659 Tasks
A React-based UI toolkit for the web
Languages
TypeScript85.2%SCSS7%MDX4.8%JavaScript2.5%Shell0.3%HTML0.2%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 55 / 59 | $5.33 | $10/$50 | 12m34s | ||
2 | Mini-SWE-agent | 52 / 59 | $0.39 | $2/$12 | 8m46s | ||
3 | Mini-SWE-agent | 52 / 59 | $3.00 | $5/$25 | 22m10s | ||
4 | Mini-SWE-agent | 52 / 59 | $3.33 | $5/$25 | 23m06s | ||
5 | Mini-SWE-agent | 52 / 59 | $0.58 | $1.4/$4.4 | 26m33s | ||
6 | Mini-SWE-agent | 51 / 59 | $5.85 | $3/$15 | 13m16s | ||
7 | Mini-SWE-agent | 51 / 59 | $0.07 | $0.2/$1.2 | 14m34s | ||
8 | Mini-SWE-agent | 51 / 59 | $0.48 | $2/$6 | 17m13s | ||
9 | Mini-SWE-agent | 51 / 59 | $3.67 | $2/$10 | 35m21s | ||
10 | Mini-SWE-agent | 50 / 59 | $1.74 | $4/$20 | 10m31s | ||
11 | Mini-SWE-agent | 50 / 59 | $1.55 | $5/$30 | 18m06s | ||
12 | Mini-SWE-agent | 50 / 59 | $1.05 | $1.5/$9 | 23m27s | ||
13 | Mini-SWE-agent | 47 / 59 | $0.54 | $2/$12 | 14m06s | ||
14 | Mini-SWE-agent | 42 / 59 | $0.27 | $1/$5 | 21m19s |
Key Takeaways
- GPT-5.6 Terra with Mini-SWE-agent matches Claude Opus 4.7, Claude Opus 4.8, and GLM 5.2 (Fireworks) at 52 of 59 tasks.
- GPT-5.6 Luna with Mini-SWE-agent resolves 51 of 59 tasks at $0.07 per test, while GPT-5.6 Sol resolves 50 at $1.74 per test.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
Claude Opus 4.7
Claude Opus 4.8
Claude Fable 5
GLM 5.2
Gemini 3.5 Flash
Kimi K3
GPT-5.6 Luna
GPT-5.6 Sol
GPT 5.5
Claude Haiku 4.5 (Nonthinking)
Grok 4.5
Gemini 3.1 Pro Preview (02/26)
GPT-5.6 Terra
Cost is the clearest tradeoff in this comparison. Claude Fable 5 leads at 93.22% for $5.33 per test. GPT-5.6 Luna is the lower-cost option at 86.44% for $0.07 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 5
GLM 5.2
Gemini 3.5 Flash
Claude Opus 4.8
Claude Opus 4.7
Claude Haiku 4.5 (Nonthinking)
GPT 5.5
Grok 4.5
GPT-5.6 Luna
Gemini 3.1 Pro Preview (02/26)
Kimi K3
Claude Fable 5
GPT-5.6 Sol
GPT-5.6 Terra
Latency separates several models with similarly strong scores. Claude Fable 5 leads at 93.22%, while GPT-5.6 Terra is fastest at 8m 46s with 88.14% accuracy.