Archived Benchmark

Code Migration v1.0 has been superseded by Code Migration v1.1, which runs on the Pi agent harness with revised test suites and replaces it in the Coding sector of Vals Index v2.2. Previous results are preserved here for posterity.

Proprietary

Code Migration

Updated 10/29/2026

Can language models reimplement working programs in another language?

As of October 29, 2026, Claude Sonnet 5.5 ranks first on Code Migration v1 with 69.83%, followed by Gemini 4 Argon (68.17%) and GPT-6 Astra (67.74%).

Code MigrationRewriting programs in new languages
Code Migration•Oct 29, 2026
ACCURACY
Lab:
Pinch or Ctrl/Cmd + scroll to zoom. Once zoomed, scroll or swipe with two fingers to pan, or drag to pan. Shift + drag draws a box to zoom into; turn on box zoom to draw one without Shift. Plus and minus keys zoom, arrow keys pan, 0 resets.

Code Migration leaderboard

Rank Model Accuracy Cost / Test Tokens / Test Duration / Test
1 Claude Sonnet 5.5 69.83% $75.83 351.92m 3h20m
2 Gemini 4 Argon 68.17% $57.82 154.36m 2h12m
3 GPT-6 Astra 67.74% $44.36 19.23m 58m30s
4 Claude Opus 5.5 66.65% $112.97 263.80m 3h19m
5 GPT-6.1 Sol 65.12% $6.51 19.43m 1h36m
6 Wonder 62.55% $69.54 209.31m 2h51m
7 Wonder Full 61.03% $38.41 175.58m 3h53m
8 Claude Opus 5 57.47% $60.51 107.67m 2h51m
9 GPT-6 Sol 57.20% $15.68 45.67m 1h26m
10 Claude Fable 5 55.06% $112.10 104.61m 1h49m
11 Claude Fable 5.1 54.61% $70.97 179.67m 4h20m
12 GPT-5.6 Sol 52.92% $24.54 35.28m 59m12s
13 GPT-5.6 Terra 47.80% $8.13 29.05m 48m34s
14 Hy4 Preview 47.43% $3.41 65.75m 2h13m
15 Muse Spark 1.3 Max 47.41% $14.21 96.41m 1h24m
16 Claude Opus 4.8 47.25% $30.51 47.22m 1h15m
17 DeepSeek V4.1 Flash 45.62% $0.94 75.85m 1h45m
18 GPT 5.5 45.16% $6.44 8.11m 31m35s
19 Grok 4.7 44.82% $36.55 40.75m 1h09m
20 Grok 4.6 44.57% $14.58 29.54m 1h30m
21 Claude Haiku 5.5 44.57% $6.23 102.33m 1h23m
22 GPT-5.6 Luna 44.55% $1.88 71.35m 58m35s
23 Claude Sonnet 5 44.39% $35.31 169.64m 1h57m
24 GLM 5.3 44.22% $24.91 93.79m 3h47m
25 DeepSeek V4.1 Flash 1017 44.21% $1.86 245.85m 2h11m
26 Claude Opus 4.7 43.88% $36.39 63.64m 1h12m
27 MiMo V2.6 Pro 43.01% $0.69 61.67m 2h49m
28 GPT-6 Luna 42.55% $0.60 34.65m 50m23s
29 DeepSeek V4 Pro 0813 41.54% $18.58 177.45m 3h32m
30 MiMo V2.6 Flash 40.93% $0.49 112.54m 3h12m
31 Claude Sonnet 4.6 39.89% $38.51 129.44m 1h40m
32 Step 5 Preview 38.98% $7.61 139.25m 3h34m
33 DeepSeek V4 Flash 0731 38.63% $3.61 142.34m 2h08m
34 GLM 5.2 37.87% $12.64 46.21m 1h54m
35 Grok 4.5 36.60% $3.37 7.79m 23m29s
36 Gemini 3.8 Flash 36.55% $18.49 95.73m 2h37m
37 GPT 5.4 (xhigh) 34.98% $7.76 17.94m 35m46s
38 Gemini 3.7 Flash 34.80% $21.46 92.69m 2h39m
39 Ember-1 32.51% $51.11 150.87m 4h09m
40 Muse Spark 1.1 31.11% $4.56 29.70m 39m11s
41 Gemini 3.6 Flash 30.93% $8.93 42.72m 1h30m
42 Mistral Large 4 30.56% $24.52 93.31m 4h51m
43 Muse Spark 1.2 29.95% $3.78 20.13m 31m07s
44 Kimi K2.6 27.77% $5.11 30.36m 1h30m
45 Muse Spark 1.3 27.58% $4.62 21.89m 1h08m
46 Gemini 3.5 Flash 26.75% $5.73 21.14m 19m28s
47 DeepSeek V4 26.20% $1.07 13.76m 28m58s
48 GLM 5.1 25.77% $5.70 24.46m 58m22s
49 Kimi K2.7 Code 25.39% $5.04 26.60m 1h39m
50 Qwen 3.8 Max 23.96% $10.54 38.10m 4h28m
51 MiMo V2.5 Pro 21.56% $0.24 19.60m 48m53s
52 Muse Spark Open 21.17% $9.04 20.48m 1h53m
53 GLM 5.3 Flash 20.52% $3.13 214.21m 3h26m
54 MiniMax-M3 19.93% $7.07 59.92m 1h14m
55 Gemini 3.1 Pro Preview (02/26) 17.31% $1.61 3.17m 12m01s
56 Kimi K3 16.10% $13.87 34.75m 4h18m
57 GPT 5.4 Nano 14.47% $0.42 10.04m 26m34s
58 MiMo V2.5 14.23% $0.10 17.86m 37m55s
59 Qwen 3.8 27B 14.16% $21.68 44.88m 4h46m
60 Inkling Small 13.68% $0.60 8.14m 20m33s
61 Qwen 3.7 Max 13.07% $1.93 3.77m 25m32s
62 GPT 5.4 Mini 12.94% $1.95 14.54m 28m53s
63 Qwen 3.7 Plus 12.86% $0.42 4.65m 35m47s
64 Inkling 11.79% $3.01 13.44m 29m34s
65 Qwen 3.6 Plus 11.10% $4.78 9.81m 29m50s
66 MiniMax-M2.7 8.69% $0.59 11.34m 20m59s
67 Claude Haiku 4.5 (Thinking) 7.55% $0.98 5.40m 9m54s
68 Kimi K2.5 6.96% $0.93 7.00m 1h04m
69 Grok 4.3 6.79% $0.42 968.09k 3m00s
70 Gemini 3 Flash (12/25) 6.39% $0.34 2.15m 4m44s
71 Gemini 3.5 Flash Lite 6.15% $0.53 3.29m 22m02s
72 Mistral Medium 3.5 5.13% $18.62 11.64m 3h38m
73 Nemotron 3 Ultra 4.91% - 17.78m 1h31m
74 Gemini 3.1 Flash Lite Preview 4.61% $0.05 579.96k 92.63s
75 Mercury 2.5 4.52% $0.63 7.43m 16m14s
76 Nemotron 3.5 Lightning 2.95% $0.27 24.05m 43m25s
77 Ling 3.0 Flash Fin 1.82% $0.10 4.98m 29m30s
78 Grok 4.20 (Reasoning) 0.31% $0.29 568.20k 6m14s
79 Ling 3.0 Flash 0.00% $0.06 1.72m 4h59m

Background

Translating a working program from one language to another is a costly, high-stakes engineering task, whether porting a tool to a new ecosystem or modernizing legacy systems written in languages that are increasingly hard to staff.

COBOL-to-Java migration carries real economic weight: decades-old COBOL still runs core transaction systems at many of the world’s largest banks, insurers, and government agencies. These institutions are spending heavily on multi-year programs to modernize that code onto maintainable languages like Java. The work is slow, risky, and exactly the kind of task reliable automation could transform.

Migration between modern languages matters too, driven by the pursuit of better runtime performance, richer tooling and ecosystems, or the memory-safety guarantees of a language like Rust.

Code Migration measures how well today’s frontier models can automate both kinds of work (modernizing legacy COBOL to Java, and translating between modern languages) by taking an existing program and faithfully reproducing its behavior in the target language.

Results

Accuracy vs. Cost

Code Migration

Cost and accuracy trade off sharply across the leaderboard. GPT-6 Astra leads at 67.7% for about $44 per test; Claude Opus 5 follows at 57.5% for about $61 per test, while Claude Fable 5 reaches 55.1% at about $112 per test. DeepSeek V4.1 Flash has the best accuracy per dollar of any model above 30%: 45.6% for under $1 per test, ahead of Claude Sonnet 4.6 (39.9%) at roughly a fortieth of its cost. GPT 5.5 reaches 45% at about $6 per test, nearly matching Claude Opus 4.8 (47%) for roughly a fifth of the cost, and ahead of the pricier Claude Opus 4.7. Further down, DeepSeek V4 reaches about 26% for about $1 per test, and MiMo V2.5 Pro about 22% for under $0.25.

Correctness vs. Code Quality

Code Quality vs Accuracy
Code qualityAccuracy
Code quality is the mean of four reviewer dimensions (readability, documentation, file structure, language idioms) over CLI tasks, scored independently of hidden-test pass rateHover a point for the model

Passing hidden tests and writing clean code are related but distinct: the strongest test-passers are not always the highest-rated implementations.

Difficulty by Target Language

CLI Accuracy by Target Language
Best target per model in darker greenAll 79 models
Model
Python
Java
Kotlin
Rust
C++
CLI avg
1Claude Sonnet 5.5
70.6
71.5
67.7
70.7
68.2
69.8
2Gemini 4 Argon
68.5
76.6
68.1
55.6
68.5
67.6
3GPT-6 Astra
65.3
68.7
66.9
68.8
67.9
67.5
4Claude Opus 5.5
67.9
65.6
68.8
59.5
64.6
65.5
5GPT-6.1 Sol
55.4
69.8
67.0
60.7
63.6
63.5
6Claude Fable 5
64.5
55.7
62.6
56.8
60.8
60.1
7Wonder
58.8
63.1
57.7
61.2
59.8
60.1
8Wonder Full
63.5
61.0
57.0
50.3
58.4
58.0
9Claude Opus 5
53.2
55.6
51.5
53.7
52.4
53.3
10GPT-6 Sol
52.2
54.2
51.2
53.4
54.6
52.9
11Claude Fable 5.1
54.5
53.3
51.8
57.8
43.2
52.8
12GPT-5.6 Sol
47.9
49.8
48.4
40.7
49.7
47.2
13GPT-5.6 Terra
38.2
42.8
42.0
41.1
35.7
40.4
14Muse Spark 1.3 Max
42.9
40.6
39.8
40.7
36.4
40.4
15Claude Opus 4.8
43.8
37.4
40.6
36.6
43.6
40.1
16Hy4 Preview
41.2
37.9
41.6
39.4
38.7
39.9
17Grok 4.6
41.1
39.8
39.1
39.2
37.2
39.4
18DeepSeek V4.1 Flash
35.1
38.2
42.4
35.1
34.4
37.5
19GPT-6 Luna
36.6
39.2
37.6
37.6
33.5
37.2
20GPT 5.5
38.1
33.8
37.9
37.6
37.0
36.9
21Grok 4.7
37.0
38.8
32.4
39.2
35.1
36.4
22Claude Sonnet 5
35.6
40.2
37.2
35.5
30.4
36.3
23Claude Haiku 5.5
36.1
35.4
36.4
38.3
33.1
36.1
24GPT-5.6 Luna
33.3
38.1
36.3
36.9
35.5
36.1
25GLM 5.3
39.8
32.4
41.0
34.2
26.0
35.6
26DeepSeek V4.1 Flash 1017
45.7
39.9
38.7
26.6
19.6
35.6
27Claude Opus 4.7
37.5
34.8
33.9
35.2
34.5
35.2
28MiMo V2.6 Pro
35.0
39.5
27.1
36.0
33.4
34.0
29DeepSeek V4 Pro 0813
37.2
30.5
32.2
30.7
28.0
32.1
30DeepSeek V4 Flash 0731
33.6
35.4
29.4
30.4
30.9
32.0
31Step 5 Preview
34.1
32.8
33.1
36.7
17.2
32.0
32MiMo V2.6 Flash
28.4
37.3
29.0
34.2
25.5
31.2
33Claude Sonnet 4.6
33.2
29.6
28.4
29.4
32.1
30.3
34GLM 5.2
23.2
26.7
29.7
27.2
29.6
27.2
35Gemini 3.8 Flash
23.0
22.0
28.7
33.4
19.6
25.9
36Grok 4.5
25.2
25.8
24.3
24.9
28.5
25.5
37GPT 5.4 (xhigh)
22.1
26.2
22.9
25.7
21.3
23.8
38Gemini 3.7 Flash
21.6
23.3
20.8
30.3
21.8
23.5
39Ember-1
33.8
19.5
23.9
21.8
14.0
23.3
40Gemini 3.6 Flash
16.1
21.8
19.9
20.1
17.9
19.3
41GLM 5.1
17.7
17.1
20.2
16.3
25.0
18.8
42Muse Spark 1.1
15.0
20.6
21.3
22.4
9.9
18.6
43Kimi K2.6
19.5
20.0
15.9
14.7
21.4
18.0
44Mistral Large 4
23.4
17.0
14.1
18.4
13.3
17.4
45Gemini 3.5 Flash
20.6
11.3
18.5
16.3
20.2
17.2
46Muse Spark 1.2
16.9
20.2
16.5
20.7
7.3
17.1
47DeepSeek V4
20.2
17.1
13.3
13.7
12.0
15.5
48Qwen 3.6 Plus
15.6
14.7
15.1
11.3
15.3
14.4
49MiMo V2.5 Pro
11.1
16.9
10.8
16.1
16.7
14.0
50Muse Spark 1.3
14.0
16.3
16.8
10.4
9.4
13.9
51Kimi K2.7 Code
12.6
13.7
14.3
14.8
10.6
13.4
52Qwen 3.8 Max
14.1
11.7
10.4
13.5
9.4
11.9
53MiniMax-M3
13.3
11.9
13.7
10.9
7.7
11.9
54Qwen 3.7 Plus
9.1
12.9
13.1
11.4
12.9
11.9
55Qwen 3.7 Max
11.3
11.7
12.9
10.1
11.8
11.6
56Gemini 3.1 Pro Preview (02/26)
12.6
10.1
10.2
11.3
9.6
10.8
57Muse Spark Open
12.5
11.4
13.5
7.4
3.4
10.4
58MiniMax-M2.7
11.1
13.9
7.2
9.3
9.3
10.2
59Claude Haiku 4.5 (Thinking)
7.4
12.6
10.5
11.3
7.4
10.1
60MiMo V2.5
11.2
13.8
10.5
5.0
7.3
9.9
61Grok 4.3
10.9
7.2
10.0
9.0
7.4
9.1
62Gemini 3 Flash (12/25)
7.2
7.5
9.0
5.6
16.3
8.5
63GPT 5.4 Mini
11.5
7.5
8.1
7.1
6.0
8.2
64Kimi K2.5
6.5
9.9
7.6
6.4
10.8
8.0
65GLM 5.3 Flash
8.9
5.1
7.1
11.0
3.4
7.4
66Inkling
7.4
9.1
6.7
9.1
0.3
7.0
67Mercury 2.5
4.7
8.9
7.2
5.5
1.6
6.0
68Inkling Small
7.4
7.0
4.5
6.7
3.2
5.9
69Gemini 3.5 Flash Lite
5.3
4.5
6.3
9.7
2.4
5.9
70Gemini 3.1 Flash Lite Preview
3.2
8.4
5.8
4.2
7.6
5.7
71Mistral Medium 3.5
3.4
6.5
5.7
5.8
2.1
5.0
72Nemotron 3 Ultra
6.0
5.1
5.8
2.0
3.4
4.6
73Nemotron 3.5 Lightning
4.1
4.4
3.6
1.4
7.7
3.9
74GPT 5.4 Nano
5.3
3.3
2.5
3.5
2.0
3.4
75Qwen 3.8 27B
2.6
2.4
0.8
3.9
1.5
2.2
76Kimi K3
0.5
0.7
2.3
1.3
3.0
1.5
77Ling 3.0 Flash Fin
0.4
3.3
0.5
0.7
0.0
1.1
78Grok 4.20 (Reasoning)
0.5
0.5
0.4
0.3
0.5
0.4
79Ling 3.0 Flash
0.0
0.0
0.0
0.0
0.0
0.0
Bars share one scale, 0 to 80% pass rate; rows ranked by CLI averageEach of the 30 source repositories is migrated into the four non-source target languages

The chart groups hidden-test pass rate by target language, with one bar per model in each group. Overall difficulty is tightly clustered: mean pass rates range from 21.90% on Rust to 23.13% on Java, with Python (22.94%), Kotlin (22.56%), and C++ (22.15%) between them. The strongest target varies by model; Opus 5, for example, performs best on Java at 55.58%.

Behavioral Test Pass-Rate Distribution

Share of tasks at or above a hidden-test pass threshold
Each line is one model: the share of its tasks that pass at least the given fraction of hidden testsClick a legend entry to toggle a model; double-click to isolate it

This graph shows how models perform at different hidden-test pass-rate thresholds. A threshold of 80%, for example, counts a task as resolved only if the rebuilt program passes at least 80% of its hidden behavior tests; a threshold of 100% requires reproducing the tested behavior exactly. The steeper a model’s drop-off near the high thresholds, the more all-or-nothing its migrations.

On the CLI split, GPT-6 Astra clears at least half of a task’s hidden tests on 89 of the 120 migrations, followed by Claude Fable 5 on 73, Claude Opus 5.5 on 65 and Claude Opus 5 on 64. Astra keeps its lead at higher thresholds: 78 tasks pass at least 65% of their hidden tests, compared with 64 for Fable 5, 51 for Opus 5.5 and 48 for Opus 5. Fable 5 is still the only model to fully reproduce any CLI task, resolving 6 of 120.

Methodology

Every task runs the same way. The model works in an offline sandbox with the source program and task instructions, and submits a target-language implementation plus a Dockerfile. The grader builds that image and runs a hidden behavior-test suite the model never sees. Before scoring, a separate anti-cheat checker inspects each submission and zeroes any that wrap the original executable, copy reference artifacts, or hardcode expected outputs. It also verifies that the submission is actually written in the requested target language, so a model cannot pass by leaving the program in its source language.

The CLI split is 30 open-source repositories spanning Python, Java, Kotlin, Rust, and C++. Each is migrated into its four non-source target languages (120 scored repository-language migrations in all), and every migration is graded against a hidden suite of roughly 100 to 400 behavior tests, exercised through the program’s command-line interface. The COBOL split is a 10-repository COBOL-to-Java test set (a mix of open-source COBOL programs and synthetic operational systems built for the benchmark) that stresses file formats, batch workflows, terminal interfaces, and persistent state.

The headline score is hidden-test pass rate. A CLI repository’s score is the average across its four target languages, so all 30 repositories count equally despite the language fan-out. The Overall score keeps every source repository equally weighted, which gives the CLI split (30 repositories) three times the weight of COBOL (10). Code quality is reported as a separate diagnostic (a reviewer rating of readability, documentation, structure, and idiomatic style) and never affects the pass-rate score.