Mercury 2.5 vs GPT 5.4 Mini: Benchmark Comparison
Mercury 2.5 has the higher score on 0 of 4 shared benchmarks; GPT 5.4 Mini leads on 4.
The largest observed score gap is 44.29 pts on Vibe Code Bench v1.1 , where GPT 5.4 Mini leads.
Reported ±1 standard-error ranges overlap on 0 of 4 shared benchmarks with comparable uncertainty data. This is not a pairwise statistical significance test.
Shared benchmark results
Scores and reported standard errors come from the latest visible Vals benchmark version available for both models.
| Benchmark | Mercury 2.5 | GPT 5.4 Mini | Gap | Reported uncertainty |
|---|---|---|---|---|
| Legal Research Bench | 4.33% ±1.41 | 12.50% ±2.30 | 8.17 pts | Reported ±1 SE ranges do not overlap |
| Tax Agent Bench | 2.49% ±1.02 | 8.81% ±2.04 | 6.32 pts | Reported ±1 SE ranges do not overlap |
| Terminal-Bench 4.0 | 0.00% ±0.00 | 2.52% ±1.01 | 2.52 pts | Reported ±1 SE ranges do not overlap |
| Vibe Code Bench v1.1 | 3.68% ±1.41 | 47.97% ±5.61 | 44.29 pts | Reported ±1 SE ranges do not overlap |
Performance by category
| Category | Mercury 2.5 average | GPT 5.4 Mini average |
|---|---|---|
| Legal | 4.33% | 12.50% |
| Finance | 2.49% | 8.81% |
| Coding | 1.84% | 25.25% |
Cost and latency
Cost per test appears only where the benchmark reports it for both models. Latency is the measured completion time for that benchmark.
| Benchmark | Mercury 2.5 cost | GPT 5.4 Mini cost | Mercury 2.5 latency | GPT 5.4 Mini latency |
|---|---|---|---|---|
| Legal Research Bench | $0.06 | $1.86 | 47.61s | 37m36s |
| Tax Agent Bench | $0.02 | $0.81 | 29.67s | 16m46s |
| Terminal-Bench 4.0 | $0.37 | $1.44 | 15m14s | 29m30s |
| Vibe Code Bench v1.1 | $0.38 | $1.19 | 3m13s | 34m15s |
Results available only for Mercury 2.5
- LegalBench
- MedCode
- MedScribe
- ProofBench v1.1
- Terminal-Bench-Science
- IOI
- SkillsBench
- CyberBench v1.1
- Public Benefits Bench v1.1
Results available only for GPT 5.4 Mini
- MortgageTax
- TaxEval v2
- GPQA Diamond
- MMLU Pro
- MMMU Pro
- SAGE
- LiveCodeBench
- ProgramBench
- SWE-bench