Ember-1 vs Mistral Large 4: Benchmark Comparison
Ember-1 has the higher score on 6 of 8 shared benchmarks; Mistral Large 4 leads on 1.
The largest observed score gap is 51.00 pts on ProofBench v1.1 , where Ember-1 leads.
Reported ±1 standard-error ranges overlap on 4 of 8 shared benchmarks with comparable uncertainty data. This is not a pairwise statistical significance test.
Shared benchmark results
Scores and reported standard errors come from the latest visible Vals benchmark version available for both models.
| Benchmark | Ember-1 | Mistral Large 4 | Gap | Reported uncertainty |
|---|---|---|---|---|
| Legal Research Bench | 44.71% ±3.46 | 31.73% ±3.23 | 12.98 pts | Reported ±1 SE ranges do not overlap |
| Tax Agent Bench | 31.08% ±3.34 | 30.39% ±3.31 | 0.69 pts | Reported ±1 SE ranges overlap |
| ProofBench v1.1 | 61.00% ±4.90 | 10.00% ±3.02 | 51.00 pts | Reported ±1 SE ranges do not overlap |
| BioMysteryBench | 67.04% ±3.03 | 67.04% ±2.67 | 0.00 pts | Reported ±1 SE ranges overlap |
| MysteryMechanism | 22.52% ±2.81 | 12.16% ±2.20 | 10.36 pts | Reported ±1 SE ranges do not overlap |
| IOI | 46.33% ±9.16 | 45.28% ±3.78 | 1.05 pts | Reported ±1 SE ranges overlap |
| Terminal-Bench 4.0 | 20.71% ±3.64 | 27.78% ±1.82 | 7.07 pts | Reported ±1 SE ranges do not overlap |
| Vibe Code Bench v1.1 | 83.87% ±3.15 | 78.40% ±3.55 | 5.47 pts | Reported ±1 SE ranges overlap |
Performance by category
| Category | Ember-1 average | Mistral Large 4 average |
|---|---|---|
| Legal | 44.71% | 31.73% |
| Finance | 31.08% | 30.39% |
| Math | 61.00% | 10.00% |
| Science | 44.78% | 39.60% |
| Coding | 50.30% | 50.49% |
Cost and latency
Cost per test appears only where the benchmark reports it for both models. Latency is the measured completion time for that benchmark.
| Benchmark | Ember-1 cost | Mistral Large 4 cost | Ember-1 latency | Mistral Large 4 latency |
|---|---|---|---|---|
| Legal Research Bench | $3.67 | $5.33 | 38m34s | 1h33m |
| Tax Agent Bench | $2.97 | $2.72 | 30m46s | 57m18s |
| ProofBench v1.1 | $1.74 | $0.64 | 21m09s | 20m59s |
| BioMysteryBench | $0.91 | $2.07 | 9m26s | 25m32s |
| MysteryMechanism | $1.29 | $2.10 | 20m15s | 35m37s |
| IOI | $24.46 | $5.39 | 1h47m | 1h24m |
| Terminal-Bench 4.0 | $19.48 | $10.25 | 1h55m | 1h40m |
| Vibe Code Bench v1.1 | $8.56 | $39.46 | 39m18s | 2h53m |
Results available only for Ember-1
- Vals Science Index
- Terminal-Bench-Science
- CyberBench v1.1
- Public Benefits Bench v1.1
Results available only for Mistral Large 4
- Vals Finance Index
- Vals Legal Index
- HLAB-Vals-1.1
- EMB v1.1
- Finance Agent v2.1
- MedCode
- MedScribe
- SWE-bench Multimodal
- Vibe Code Bench 1-100