Proprietary

MedCode

Updated 10/29/2026

Can models support the medical billing process?

As of October 29, 2026, Claude Opus 5 ranks first on MedCode with 63.57%, followed by Gemini 3.1 Pro Preview (02/26) at 59.06% and Gemini 4 Argon (58.80%).

MedCodeMedical billing code support
MedCode•Oct 29, 2026
ACCURACY
Lab:
Pinch or Ctrl/Cmd + scroll to zoom. Once zoomed, scroll or swipe with two fingers to pan, or drag to pan. Shift + drag draws a box to zoom into; turn on box zoom to draw one without Shift. Plus and minus keys zoom, arrow keys pan, 0 resets.

MedCode leaderboard

Rank Model Accuracy Cost In / Out Tokens / Test Latency
1 Claude Opus 5 63.57% $5 / $25 19.09k 24.50s
2 Gemini 3.1 Pro Preview (02/26) 59.06% $2 / $12 11.68k 38.52s
3 Gemini 4 Argon 58.80% $2 / $10 16.11k 79.53s
4 Claude Fable 5 56.07% $10 / $50 24.58k 91.44s
5 Gemini 3 Flash (12/25) 55.92% $0.5 / $3 11.69k 44.15s
6 Gemini 3.5 Flash 55.83% $1.5 / $9 15.77k 25.29s
7 Claude Opus 4.7 54.86% $5 / $25 21.88k 54.25s
8 Claude Fable 5.1 53.51% $10 / $50 35.16k 3m35s
9 Gemini 3.7 Flash 53.39% $1.5 / $7.5 14.35k 9.16s
10 Claude Opus 4.8 53.22% $5 / $25 26.78k 105.92s
11 Gemini 3.6 Flash 53.15% $0.75 / $3.75 15.13k 16.68s
12 Claude Sonnet 5.5 52.92% $2 / $10 51.98k 3m59s
13 GPT 5.1 52.73% $1.25 / $10 10.93k 54.55s
14 Gemini 3 Pro (11/25) 52.20% $2 / $12 11.98k 55.84s
15 Claude Haiku 5.5 51.54% $0.1 / $0.5 - 3m46s
16 Muse Spark 51.31% - 13.54k 2m03s
17 Gemini 2.5 Pro 50.59% $1.25 / $10 11.64k 27.41s
18 Wonder Full 49.93% $4 / $20 - 3m03s
19 Claude Opus 5.5 49.80% $4 / $20 45.87k 4m07s
20 GPT 5.2 49.75% $1.75 / $14 10.89k 2m37s
21 GPT 5 49.63% $1.25 / $10 14.08k 58.27s
22 Grok 4.7 49.55% $2 / $6 25.82k 3m33s
23 Kimi K3 49.36% $3 / $15 16.61k 36.99s
24 Muse Spark 1.2 49.35% $1.25 / $4.25 16.78k 60.17s
25 Claude Opus 4.5 (Thinking) 49.16% $5 / $25 4.36k 60.84s
26 Claude Opus 4.6 (Thinking) 49.13% $5 / $25 10.28k 2m36s
27 GPT 5.5 49.10% $5 / $30 18.45k 2m40s
28 GPT-6.1 Sol 48.84% $2 / $10 16.61k 115.34s
29 GPT-6 Astra 48.49% $10 / $50 17.71k 102.86s
30 Claude Opus 4.6 (Nonthinking) 48.24% $5 / $25 801 4.06s
31 Gemini 3.8 Flash 48.13% $0.75 / $3.75 27.59k 42.89s
32 Gemini 3.1 Flash Lite Preview 47.60% $0.25 / $1.5 13.74k 8.55s
33 Claude Sonnet 5 47.54% $2 / $10 31.20k 2m15s
34 o3 47.29% $2 / $8 11.87k 17.68s
35 Claude Opus 4.1 (Thinking) 47.23% $15 / $75 13.47k 33.26s
36 GPT-6 Sol 47.07% $2 / $10 16.72k 62.71s
37 MiniMax-M3 46.29% $0.3 / $1.2 13.15k 63.12s
38 Claude Opus 4.5 (Nonthinking) 45.17% $5 / $25 827 5.02s
39 MiMo V2.6 Pro 44.97% $0.435 / $0.87 20.49k 2m57s
40 Grok 4.6 44.71% $2 / $6 15.52k 2m07s
41 GPT-6 Luna 44.69% $0.1 / $0.5 22.79k 108.77s
42 Claude Sonnet 4.5 (Thinking) 44.13% $3 / $15 16.78k 74.33s
43 GPT-5.6 Sol 43.97% $4 / $20 18.80k 96.59s
44 Gemini 3.5 Flash Lite 43.49% $0.3 / $2.5 13.40k 6.02s
45 GPT-5.6 Terra 43.41% $2 / $12 12.92k 18.41s
46 Grok 4.5 43.29% $2 / $6 15.17k 59.64s
47 Hy4 Preview 43.25% $0.834 / $2.501 32.32k 6m39s
48 GPT 5 Mini 43.05% $0.25 / $2 12.28k 28.17s
49 GLM 5.3 42.86% $1.4 / $4.4 25.65k 2m49s
50 DeepSeek V4 Pro 0813 42.47% $1.32 / $3.96 22.63k 3m01s
51 GPT-5.6 Luna 42.39% $0.2 / $1.2 22.26k 81.28s
52 GLM 5.1 41.60% $1 / $3.2 15.29k 77.58s
53 DeepSeek V4 Flash 0731 41.41% $0.3 / $1.2 25.45k 2m07s
54 Claude Opus 4.1 (Nonthinking) 41.37% $15 / $75 12.61k 13.08s
55 GPT 5.4 (xhigh) 41.29% $2.5 / $15 23.29k 3m07s
56 Inkling 41.19% $1 / $4.05 20.11k 2m47s
57 DeepSeek V4.1 Flash 41.17% $0.3 / $1.2 18.44k 35.13s
58 MiMo V2.6 Flash 41.06% $0.14 / $0.28 16.69k 93.37s
59 GPT 5.4 Nano 41.03% $0.2 / $1.25 11.61k 8.04s
60 GLM 5.2 40.77% $1.4 / $4.4 17.81k 95.27s
61 Mistral Large 4 Preview 40.69% $0.7 / $2.1 17.53k 68.68s
62 Qwen 3.8 Max 40.67% $2 / $6 28.33k 6m18s
63 Claude Sonnet 4.5 (Nonthinking) 40.57% $3 / $15 12.81k 12.01s
64 Gemini 2.5 Flash Preview (9/25) (Nonthinking) 40.54% $0.3 / $2.5 11.64k 12.70s
65 DeepSeek V4 40.45% $1.32 / $3.96 22.60k 6m23s
66 Gemini 2.5 Flash (7/17) (Thinking) 40.36% $0.3 / $2.5 11.63k 22.83s
67 Gemini 2.5 Flash Preview (9/25) (Thinking) 40.33% $0.3 / $2.5 11.63k 16.11s
68 Kimi K2.6 40.14% $0.95 / $4 18.70k 5m05s
69 Kimi K2.5 39.32% $0.6 / $3 14.61k 75.45s
70 Qwen 3.7 Max 38.75% $2.5 / $7.5 13.64k 28.75s
71 Nemotron 3 Ultra 38.62% - 14.17k 18.68s
72 Gemini 2.5 Flash (7/17) (Nonthinking) 38.42% $0.3 / $2.5 11.64k 22.14s
73 Grok 4 38.08% $3 / $15 11.17k 89.00s
74 Grok 4.3 38.07% $1.25 / $2.5 14.11k 43.92s
75 Inkling Small 37.89% $0.3 / $1.2 22.05k 3m11s
76 Grok 4 Fast (Reasoning) 37.38% $0.2 / $0.5 10.64k 17.52s
77 Qwen 3.6 Plus 36.89% $0.5 / $3 15.21k 56.22s
78 Llama 4 Maverick 36.51% $0.22 / $0.88 11.36k 21.24s
79 Claude Sonnet 4 (Thinking) 34.96% $3 / $15 14.36k 39.80s
80 MiniMax-M2.7 34.44% $0.3 / $1.2 13.47k 30.25s
81 Gemini 2.5 Flash Lite (9/25) (Thinking) 34.19% $0.1 / $0.4 11.62k 10.49s
82 MiniMax-M2.1 34.08% $0.3 / $1.2 - 19.55s
83 Claude Sonnet 4 (Nonthinking) 33.94% $3 / $15 12.31k 7.30s
84 o4 Mini 33.79% $1.1 / $4.4 12.14k 21.06s
85 Mistral Medium 3.5 33.75% $1.5 / $7.5 15.91k 34.88s
86 Qwen 3.5 Flash 33.00% $0.1 / $0.4 18.83k 63.07s
87 GLM 4.7 32.77% $0.6 / $2.2 11.03k 2m03s
88 Claude Haiku 4.5 (Thinking) 32.68% $1 / $5 4.54k 34.29s
89 MiMo V2.5 Pro 32.48% $0.435 / $0.87 13.96k 38.45s
90 Ling 3.0 Flash 32.27% $0.075 / $0.22 15.30k 9.65s
91 Grok 4.20 (Reasoning) 32.16% $2 / $6 13.14k 16.55s
92 MiMo V2.5 31.89% $0.14 / $0.28 13.95k 17.70s
93 Qwen 3 VL Plus 31.65% $0.2 / $1.6 12.10k 9.95s
94 Qwen 3 Max Thinking 31.37% $1.2 / $6 12.09k 3m02s
95 Mercury 2.5 31.33% $0.2 / $0.75 14.60k 7.93s
96 GPT 5 Nano 30.44% $0.05 / $0.4 13.82k 29.74s
97 Grok 4 Fast (Non-Reasoning) 30.04% $0.2 / $0.5 10.66k 18.21s
98 Ling 3.0 Flash Fin 29.30% $0.06 / $0.18 15.43k 31.17s
99 Qwen 3.8 27B 28.70% $0.5 / $3 26.72k 89.38s
100 Grok 4.1 Fast Non-Reasoning 28.35% $0.2 / $0.5 10.65k 4.50s
101 Grok 4.1 Fast (Reasoning) 28.08% $0.2 / $0.5 10.48k 46.50s
102 Gemini 2.5 Flash Lite (Nonthinking) 27.11% $0.1 / $0.4 12.02k 5.75s
103 Gemini 2.5 Flash Lite (9/25) (Nonthinking) 27.08% $0.1 / $0.4 12.26k 6.24s
104 Llama 4 Scout 23.31% $0.18 / $0.59 11.17k 10.74s
105 Laguna M.1 23.11% - 13.62k 71.00s
106 Laguna XS.2 21.25% - 13.44k 33.09s
107 Command A+ 19.72% - 13.06k 103.75s

Partners in Evaluation

Results

Claude Opus 5 takes the top spot at 63.57%, ahead of Gemini 3.1 Pro Preview (02/26) (59.06%), Gemini 4 Argon (58.80%), Claude Fable 5 (56.07%), and Gemini 3 Flash (12/25) (55.92%).

MedCode Performance

Background

The process of accurate medical coding is extraordinarily complex. There are tens of thousands of diagnosis codes, each with detailed rules dictating sequencing, modifiers, and payer-specific requirements1. These rules vary across patient demographics, medical providers, and geographical jurisdictions, making coding extremely difficult. Any error, even those that are seemingly minor, can lead to claim denials and lost revenue for the hospital2. While AI coding systems are being adopted to improve both accuracy and finances, there is little to no oversight or evaluation for how these systems perform under real-world conditions3.

Through our MedCode benchmark, we evaluate AI systems’ ability to perform medical coding given realistic documentation constraints. In collaboration with Protege, we created a dataset of 2,755 de-identified patient records, each annotated with primary and secondary diagnosis codes, enabling coding evaluation for an entire hospitalization stay from admission through discharge.


Methodology

The International Classification of Diseases (ICD) is the global standard for classifying diseases, symptoms, and injuries as it enables consistent reporting across health systems. We evaluate AI models on the CMS ICD-10-CM standard, which is the official coding standard mandated by the Centers for Medicare & Medicaid Services (CMS), as it is publicly available with comprehensive, up-to-date documentation.

Our dataset consists of de-identified patient records containing discharge summaries supported by corresponding progress notes and consult notes throughout the duration of the patient’s hospital stay. We matched the three-digit ZIP code of each patient’s stay to geographical jurisdictions and other demographic information through reference tables, allowing us to obtain key information necessary for accurate coding without compromising patient privacy. Each sample was independently annotated by two certified professional coders, who assigned primary and secondary diagnosis codes based on the ICD-10-CM standard. They were permitted to use Codify, a professional tool for code lookup and validation, but were not allowed to use AI models. Differences between coder annotations were resolved collaboratively to ensure consistency and high quality.

Each model was given the de-identified patient records and prompted to output corresponding primary and secondary diagnoses following the ICD-10-CM standard. All models are evaluated with temperature 1, and produce at most 30k tokens.


Additional Findings

We break down model performance across several dimensions: primary versus secondary diagnoses, the most frequently occurring codes, ICD-10 chapters, and levels of code precision.

Primary vs Secondary Pass Rates

Nearly all models perform better at predicting primary diagnosis codes than secondary codes. Interestingly, very few errors stem from misclassifying a code as primary versus secondary. The majority of errors occur when models completely miss a diagnosis code that appears in the rubric.

Primary vs Secondary Pass Rates
CorrectMismatchMissing

Most Frequent Codes

No model consistently leads across the top 10 most frequently occurring diagnosis codes. This inconsistency suggests brittle generalization rather than robust clinical understanding.

Models generally perform well on physiological conditions such as type 2 diabetes (E11.9), hypertension (I10), and acute kidney failure (N17.9), but they struggle with mental-health diagnoses like F32.A (depression).

Most Frequent ICD-10 Codes Pass Rates by Model

Pass Rates by Chapter

This pattern extends to the chapter level. Models perform better on codes tied to physical conditions, while accuracy drops sharply for chapters like F (Mental, Behavioral, and Neurodevelopmental disorders) and Z (Factors influencing health status and contact with health services).

O (delivery) codes were excluded because obfuscation in the dataset made them unreliable to evaluate.

ICD-10 Chapter Pass Rates by Model

Pass Rates by Code Precision

We also investigated how models perform when diagnosis codes are evaluated with greater precision. The plot below shows pass rates at different code levels. For example, for the code F32.A, the chapter is F (mental disorders), the category is F32 (depression), and the subcategory is the full code F32.A (depression, unspecified).

The accuracy drop-off with increased precision is expected. However, even when evaluating by category alone, the best models still perform poorly, indicating that the primary cause of failure is completely missing the diagnosis, not mislabelling the specificity.

Precision Pass Rates by Model

Citations

[1] ICD.Codes. (2018, January 19). ICD-9 to ICD-10 explained. Retrieved from https://web.archive.org/web/20180119120210/https://icd.codes/articles/icd9-to-icd10-explained

[2] BlueBrix Health. (2025, March 12). The hidden costs of coding errors: How accurate medical coding boosts revenue. BlueBrix Health. https://bluebrix.health/blogs/the-hidden-costs-of-coding-errors-how-accurate-medical-coding-boosts-revenue

[3] Angus, D. C., Lee, S. I., Beam, A. L., Denecke, K., Farahani, N. I., Kohane, I. S., Matheny, M. E., Sendak, M. P., Shah, N. H., Steinhubl, S. R., & Topol, E. J. (2025). AI, health, and health care today and tomorrow: The JAMA summit report on artificial intelligence. JAMA, 333(14), 1279–1287. https://doi.org/10.1001/jama.2025.18490