Proprietary

Code Migration

Updated 10/19/2026

Can language models reimplement working programs in another language?

As of October 19, 2026, Claude Sonnet 5.5 ranks first on Code Migration with 86.88%, followed by Claude Opus 5.5 (81.01%) and GPT-6 Astra (80.58%).

Code Migration v1.1Rewriting programs in new languages
ACCURACY

Code Migration leaderboard

Rank Model Accuracy Cost / Test Input / Output Cost Duration
1 Claude Sonnet 5.5 86.88% $43.88 $2 / $10 4h07m
2 Claude Opus 5.5 81.01% $72.40 $4 / $20 3h29m
3 GPT-6 Astra 80.58% $62.07 $10 / $50 1h27m
4 GPT-6.1 Sol 77.32% $9.51 $2 / $10 1h39m
5 Wonder 72.36% $47.93 $4 / $20 2h42m
6 Gemini 4 Argon 70.80% $44.79 $4 / $20 2h22m
7 DeepSeek V4.1 Flash 1017 67.52% $1.76 $0.3 / $1.2 2h21m
8 Wonder Full 62.79% $23.58 $4 / $20 4h18m
9 DeepSeek V4.1 Flash 58.94% $1.09 $0.3 / $1.2 1h54m
10 GLM 5.3 55.07% $34.91 $1.4 / $4.4 2h58m
11 Grok 4.7 53.22% $38.05 $2 / $6 1h05m
12 Claude Haiku 5.5 52.36% $6.20 $0.1 / $0.5 1h26m
13 DeepSeek V4 Pro 0813 51.99% $12.16 $1.32 / $3.96 2h54m
14 GLM 5.3 Flash 51.65% $4.48 $0.075 / $0.25 3h20m
15 MiMo V2.6 Pro 48.79% $0.99 $0.435 / $0.87 3h28m
16 Step 5 Preview 47.00% $8.61 $1 / $2.7 3h17m
17 GPT-6 Luna 46.67% $0.83 $0.1 / $0.5 1h00m
18 Hy4 Preview 45.36% $3.58 $0.834 / $2.501 1h54m
19 MiMo V2.6 Flash 39.55% $0.53 $0.14 / $0.28 4h10m
20 Gemini 3.8 Flash 38.65% $15.79 $1.5 / $7.5 2h35m
21 Kimi K3 28.59% $35.75 $3 / $15 4h31m
22 Muse Spark Open 26.89% $3.77 $1.25 / $4.25 1h24m
23 MiniMax-M3 20.90% $8.39 $0.6 / $2.4 1h26m
24 Inkling Small 10.43% $1.36 $0.3 / $1.2 17m56s
25 Inkling 8.82% $6.13 $1 / $4.05 24m11s

What changed from v1: v1.1 runs every model on the Pi agent harness instead of mini-SWE-agent, and grades against revised hidden test suites. The tasks are the same, but scores are not directly comparable to Code Migration v1.

Background

Translating a working program from one language to another is a costly, high-stakes engineering task, whether porting a tool to a new ecosystem or modernizing legacy systems written in languages that are increasingly hard to staff.

COBOL-to-Java migration carries real economic weight: decades-old COBOL still runs core transaction systems at many of the world’s largest banks, insurers, and government agencies. These institutions are spending heavily on multi-year programs to modernize that code onto maintainable languages like Java. The work is slow, risky, and exactly the kind of task reliable automation could transform.

Migration between modern languages matters too, driven by the pursuit of better runtime performance, richer tooling and ecosystems, or the memory-safety guarantees of a language like Rust.

Code Migration measures how well today’s frontier models can automate both kinds of work (modernizing legacy COBOL to Java, and translating between modern languages) by taking an existing program and faithfully reproducing its behavior in the target language.

Results

Accuracy vs. Cost

Code Migration v1.1

Claude Sonnet 5.5 leads at 86.9% for about $44 per test, ahead of Claude Opus 5.5 (81.0%, about $72) and GPT-6 Astra (80.6%, about $62). GPT-6.1 Sol reaches 77.3% for under $10 per test, roughly a seventh of Opus 5.5’s cost. DeepSeek V4.1 Flash 1017 reaches 67.5% for under $2 per test, within about 3 points of Gemini 4 Argon (70.8%) at roughly a twenty-fifth of its cost. Further down, MiMo V2.6 Pro reaches about 49% and GPT-6 Luna about 47%, both for under $1 per test.

Correctness vs. Code Quality

Code Quality vs Accuracy
Code qualityAccuracy
Code quality is the mean of four reviewer dimensions (readability, documentation, file structure, language idioms) over CLI tasks, scored independently of hidden-test pass rateHover a point for the model

Passing hidden tests and writing clean code are related but distinct: the strongest test-passers are not always the highest-rated implementations. Claude Haiku 5.5 has the highest code-quality rating (4.12 out of 5) while passing 42.1% of CLI hidden tests.

Difficulty by Target Language

CLI Accuracy by Target Language
Best target per model in darker greenAll 25 models
Model
Python
Java
Kotlin
Rust
C++
CLI avg
1Claude Sonnet 5.5
86.2
84.1
80.9
79.3
84.4
82.8
2Claude Opus 5.5
85.7
80.3
73.9
73.9
79.0
78.3
3GPT-6 Astra
77.4
75.8
76.8
73.5
80.1
76.4
4GPT-6.1 Sol
78.0
72.7
74.1
70.7
80.8
74.8
5Gemini 4 Argon
68.0
68.7
65.7
64.6
75.7
67.8
6Wonder
69.7
66.3
69.1
59.7
72.4
67.1
7DeepSeek V4.1 Flash 1017
66.8
57.7
58.2
56.5
55.5
59.2
8Wonder Full
61.3
50.0
49.1
43.6
65.0
52.7
9DeepSeek V4.1 Flash
50.8
45.9
48.2
48.8
43.2
47.7
10GPT-6 Luna
48.6
41.9
44.8
39.9
47.8
44.3
11Claude Haiku 5.5
48.2
38.2
43.7
38.8
41.2
42.1
12GLM 5.3
49.6
43.5
40.2
37.1
26.9
40.6
13Grok 4.7
46.5
41.2
37.5
41.0
35.1
40.6
14MiMo V2.6 Pro
44.3
42.6
37.4
36.9
32.2
39.2
15GLM 5.3 Flash
44.9
43.7
36.3
32.0
35.1
38.7
16Hy4 Preview
45.5
35.5
36.6
36.9
33.6
37.9
17DeepSeek V4 Pro 0813
45.4
38.7
32.5
33.5
33.2
36.8
18Step 5 Preview
44.4
34.6
30.6
32.4
34.4
35.2
19Gemini 3.8 Flash
33.1
25.8
22.5
28.1
31.1
27.6
20MiMo V2.6 Flash
38.5
24.2
19.1
28.1
26.5
27.0
21Muse Spark Open
16.6
14.9
12.8
15.8
16.1
15.0
22Kimi K3
14.2
13.2
14.1
6.0
7.3
11.4
23MiniMax-M3
14.2
7.6
7.2
5.4
7.2
8.3
24Inkling Small
5.0
3.4
3.1
1.8
0.7
3.0
25Inkling
3.0
4.6
3.1
0.9
0.3
2.6
Bars share one scale, 0 to 90% pass rate; rows ranked by CLI averageEach of the 30 source repositories is migrated into the four non-source target languages

The chart groups hidden-test pass rate by target language, with one bar per model in each group. Mean pass rates range from 39.84% on Rust to 47.56% on Python, with Java (42.32%), C++ (41.81%), and Kotlin (40.62%) between them. The strongest target varies by model: the Claude models do best on Python (Sonnet 5.5 at 86.19%), while GPT-6 Astra and GPT-6.1 Sol do best on C++ (80.14% and 80.80%).

Behavioral Test Pass-Rate Distribution

Share of tasks at or above a hidden-test pass threshold
Each line is one model: the share of its tasks that pass at least the given fraction of hidden testsClick a legend entry to toggle a model; double-click to isolate it

This graph shows how models perform at different hidden-test pass-rate thresholds. A threshold of 80%, for example, counts a task as resolved only if the rebuilt program passes at least 80% of its hidden behavior tests; a threshold of 100% requires reproducing the tested behavior exactly. The steeper a model’s drop-off near the high thresholds, the more all-or-nothing its migrations.

On the CLI split, Claude Sonnet 5.5 clears at least half of a task’s hidden tests on 109 of the 120 migrations, followed by Claude Opus 5.5 and GPT-6 Astra on 99 each and GPT-6.1 Sol on 96. At 100%, Opus 5.5 fully reproduces 18 CLI tasks and Sonnet 5.5 15; no other model fully resolves more than 4, and GPT-6 Astra resolves none.

Methodology

Every task runs the same way. The model works in an offline sandbox with the source program and task instructions, and submits a target-language implementation plus a Dockerfile. The grader builds that image and runs a hidden behavior-test suite the model never sees. Before scoring, a separate anti-cheat checker inspects each submission and zeroes any that wrap the original executable, copy reference artifacts, or hardcode expected outputs. It also verifies that the submission is actually written in the requested target language, so a model cannot pass by leaving the program in its source language.

The CLI split is 30 open-source repositories spanning Python, Java, Kotlin, Rust, and C++. Each is migrated into its four non-source target languages (120 scored repository-language migrations in all), and every migration is graded against a hidden suite of roughly 100 to 400 behavior tests, exercised through the program’s command-line interface. The COBOL split is a 10-repository COBOL-to-Java test set (a mix of open-source COBOL programs and synthetic operational systems built for the benchmark) that stresses file formats, batch workflows, terminal interfaces, and persistent state.

The headline score is hidden-test pass rate. A CLI repository’s score is the average across its four target languages, so all 30 repositories count equally despite the language fan-out. The Overall score keeps every source repository equally weighted, which gives the CLI split (30 repositories) three times the weight of COBOL (10). Code quality is reported as a separate diagnostic (a reviewer rating of readability, documentation, structure, and idiomatic style) and never affects the pass-rate score.