Proprietary

Code Migration

Updated 11/9/2026

Can language models reimplement working programs in another language?

As of November 9, 2026, Claude Sonnet 5.5 ranks first on Code Migration with 86.88%, followed by Claude Fable 5.2 (83.27%) and Claude Opus 5.5 (81.01%).

Code Migration v1.1Rewriting programs in new languages
Code Migration v1.1•Nov 09, 2026
ACCURACY
Lab:
Pinch or Ctrl/Cmd + scroll to zoom. Once zoomed, scroll or swipe with two fingers to pan, or drag to pan. Shift + drag draws a box to zoom into; turn on box zoom to draw one without Shift. Plus and minus keys zoom, arrow keys pan, 0 resets.

Code Migration leaderboard

Rank Model Accuracy Cost / Test Tokens / Test Duration / Test
1 Claude Sonnet 5.5 86.88% $43.88 353.30m 4h07m
2 Claude Fable 5.2 83.27% $69.19 188.47m 3h07m
3 Claude Opus 5.5 81.01% $72.40 302.32m 3h29m
4 GPT-6 Astra 80.58% $62.07 31.00m 1h27m
5 GPT-6.1 Sol 77.32% $9.51 37.14m 1h39m
6 Wonder 72.36% $47.93 219.17m 2h42m
7 Gemini 4 Argon 70.80% $44.79 145.90m 2h22m
8 DeepSeek V4.1 Flash 1017 67.52% $1.76 228.26m 2h21m
9 Wonder Full 62.79% $23.58 91.28m 4h18m
10 DeepSeek V4.1 Flash 58.94% $1.09 121.32m 1h54m
11 Qwen 3.8 Max 0902 57.19% $27.08 110.10m 4h08m
12 GLM 5.3 55.07% $34.91 142.25m 2h58m
13 Grok 4.7 53.22% $38.05 46.46m 1h05m
14 Claude Haiku 5.5 52.36% $6.20 95.89m 1h26m
15 DeepSeek V4 Pro 0813 51.99% $12.16 263.33m 2h54m
16 GLM 5.3 Flash 51.65% $8.96 324.23m 3h20m
17 MiMo V2.6 Pro 48.79% $0.99 84.95m 3h28m
18 Step 5 Preview 47.00% $8.61 161.54m 3h17m
19 GPT-6 Luna 46.67% $0.83 46.81m 1h00m
20 Hy4 Preview 45.36% $3.58 61.57m 1h54m
21 MiMo V2.6 Flash 39.55% $0.53 125.33m 4h10m
22 Mercury 3 39.26% $2.11 101.88m 1h25m
23 Gemini 3.8 Flash 38.65% $15.79 96.29m 2h35m
24 Kimi K3 28.59% $35.75 114.05m 4h31m
25 Muse Spark Open 26.89% $3.77 13.30m 1h24m
26 Beam 1.0 22.85% - 9.94m 13m40s
27 MiniMax-M3 20.90% $8.39 68.44m 1h26m
28 Nemotron 3.5 Super VL 16.44% $1.60 35.60m 26m44s
29 Inkling Small 10.43% $1.36 23.15m 17m56s
30 Inkling 8.82% $6.13 18.78m 24m11s
Reasoning EffortAccuracy at three reasoning efforts per model, labelled with each provider's own setting
Code Migration v1.1•Nov 09, 2026
Accuracy
Model:
Costs are estimated from tokens. Error bars show ±1 standard error.

What changed from v1: v1.1 runs every model on the Pi agent harness instead of mini-SWE-agent, and grades against revised hidden test suites. The tasks are the same, but scores are not directly comparable to Code Migration v1.

Background

Translating a working program from one language to another is a costly, high-stakes engineering task, whether porting a tool to a new ecosystem or modernizing legacy systems written in languages that are increasingly hard to staff.

COBOL-to-Java migration carries real economic weight: decades-old COBOL still runs core transaction systems at many of the world’s largest banks, insurers, and government agencies. These institutions are spending heavily on multi-year programs to modernize that code onto maintainable languages like Java. The work is slow, risky, and exactly the kind of task reliable automation could transform.

Migration between modern languages matters too, driven by the pursuit of better runtime performance, richer tooling and ecosystems, or the memory-safety guarantees of a language like Rust.

Code Migration measures how well frontier models can automate both kinds of work (modernizing legacy COBOL to Java, and translating between modern languages) by taking an existing program and faithfully reproducing its behavior in the target language.


Results

Accuracy vs. Cost

Code Migration v1.1

Claude Sonnet 5.5 leads at 86.88%, ahead of Claude Fable 5.2 (83.27%) and Claude Opus 5.5 (81.01%).

Correctness vs. Code Quality

Code Quality vs Accuracy
Code qualityAccuracy
Code quality is the mean of four reviewer dimensions (readability, documentation, file structure, language idioms) over CLI tasks, scored independently of hidden-test pass rateHover a point for the model

Passing hidden tests and writing clean code are related but distinct: the strongest test-passers are not always the highest-rated implementations. Claude Haiku 5.5 has the highest code-quality rating.

Difficulty by Target Language

CLI Accuracy by Target Language
Best target per model in darker greenAll 30 models
Model
Python
Java
Kotlin
Rust
C++
CLI avg
1Claude Sonnet 5.5
86.2
84.1
80.9
79.3
84.4
82.8
2Claude Fable 5.2
83.4
81.1
81.1
75.1
86.1
81.0
3Claude Opus 5.5
85.7
80.3
73.9
73.9
79.0
78.3
4GPT-6 Astra
77.4
75.8
76.8
73.5
80.1
76.4
5GPT-6.1 Sol
78.0
72.7
74.1
70.7
80.8
74.8
6Gemini 4 Argon
68.0
68.7
65.7
64.6
75.7
67.8
7Wonder
69.7
66.3
69.1
59.7
72.4
67.1
8DeepSeek V4.1 Flash 1017
66.8
57.7
58.2
56.5
55.5
59.2
9Wonder Full
61.3
50.0
49.1
43.6
65.0
52.7
10DeepSeek V4.1 Flash
50.8
45.9
48.2
48.8
43.2
47.7
11Qwen 3.8 Max 0902
50.7
45.4
38.9
50.8
42.3
45.6
12GPT-6 Luna
48.6
41.9
44.8
39.9
47.8
44.3
13Claude Haiku 5.5
48.2
38.2
43.7
38.8
41.2
42.1
14GLM 5.3
49.6
43.5
40.2
37.1
26.9
40.6
15Grok 4.7
46.5
41.2
37.5
41.0
35.1
40.6
16MiMo V2.6 Pro
44.3
42.6
37.4
36.9
32.2
39.2
17GLM 5.3 Flash
44.9
43.7
36.3
32.0
35.1
38.7
18Hy4 Preview
45.5
35.5
36.6
36.9
33.6
37.9
19DeepSeek V4 Pro 0813
45.4
38.7
32.5
33.5
33.2
36.8
20Step 5 Preview
44.4
34.6
30.6
32.4
34.4
35.2
21Mercury 3
27.5
25.6
30.8
30.6
23.0
28.0
22Gemini 3.8 Flash
33.1
25.8
22.5
28.1
31.1
27.6
23MiMo V2.6 Flash
38.5
24.2
19.1
28.1
26.5
27.0
24Muse Spark Open
16.6
14.9
12.8
15.8
16.1
15.0
25Kimi K3
14.2
13.2
14.1
6.0
7.3
11.4
26Nemotron 3.5 Super VL
14.6
11.8
8.8
10.7
6.7
10.8
27Beam 1.0
12.9
9.2
7.7
4.4
11.4
8.9
28MiniMax-M3
14.2
7.6
7.2
5.4
7.2
8.3
29Inkling Small
5.0
3.4
3.1
1.8
0.7
3.0
30Inkling
3.0
4.6
3.1
0.9
0.3
2.6
Bars share one scale, 0 to 90% pass rate; rows ranked by CLI averageEach of the 30 source repositories is migrated into the four non-source target languages

The chart groups hidden-test pass rate by target language, with one bar per model in each group. The strongest target varies by model: the Claude models do best on Python, while GPT-6 Astra and GPT-6.1 Sol do best on C++.

Behavioral Test Pass-Rate Distribution

Share of tasks at or above a hidden-test pass threshold
Each line is one model: the share of its tasks that pass at least the given fraction of hidden testsClick a legend entry to toggle a model; double-click to isolate it

This graph shows how models perform at different hidden-test pass-rate thresholds. A threshold of 80%, for example, counts a task as resolved only if the rebuilt program passes at least 80% of its hidden behavior tests; a threshold of 100% requires reproducing the tested behavior exactly. The steeper a model’s drop-off near the high thresholds, the more all-or-nothing its migrations.


Methodology

Every task runs the same way. The model works in an offline sandbox with the source program and task instructions, and submits a target-language implementation plus a Dockerfile. The grader builds that image and runs a hidden behavior-test suite the model never sees. Before scoring, a separate anti-cheat checker inspects each submission and zeroes any that wrap the original executable, copy reference artifacts, or hardcode expected outputs. It also verifies that the submission is actually written in the requested target language, so a model cannot pass by leaving the program in its source language.

The CLI split is 30 open-source repositories spanning Python, Java, Kotlin, Rust, and C++. Each is migrated into its four non-source target languages (120 scored repository-language migrations in all), and every migration is graded against a hidden suite of roughly 100 to 400 behavior tests, exercised through the program’s command-line interface. The COBOL split is a 10-repository COBOL-to-Java test set (a mix of open-source COBOL programs and synthetic operational systems built for the benchmark) that stresses file formats, batch workflows, terminal interfaces, and persistent state.

The headline score is hidden-test pass rate. A CLI repository’s score is the average across its four target languages, so all 30 repositories count equally despite the language fan-out. The Overall score keeps every source repository equally weighted, which gives the CLI split (30 repositories) three times the weight of COBOL (10). Code quality is reported as a separate diagnostic (a reviewer rating of readability, documentation, structure, and idiomatic style) and never affects the pass-rate score.