We’re releasing MedScribe, a new benchmark evaluating AI systems on clinical SOAP note generation, built in collaboration with Harvard Medical School and Protege.

• GPT 5.1 takes the lead, exceeding 88% accuracy. The Claude family also demonstrates strong performance.
• Results suggest models are capable of producing structurally sound, clinically relevant medical notes and can meaningfully assist scribes in their workflow.
• Most models perform marginally worse on the Plan section, where even small errors can affect care continuity and billing accuracy.

We’re releasing MedCode, a new benchmark evaluating AI systems on medical coding from realistic patient documentation, built in collaboration with Harvard Medical School and Protege.

• Models barely exceed 55% accuracy on this critical healthcare task, with Gemini 3 Flash (12/25) taking the lead at 55.9%.
• Models perform better on physical conditions like diabetes and hypertension but struggle significantly with mental health diagnoses.
• Even at the category level, models frequently miss diagnoses entirely rather than mislabelling specificity.

As we finish evaluating Qwen 3.5 Plus, here are our key takeaways:

• The model performs well on knowledge benchmarks, ranking #6 on GPQA, #11 on MedQA and #9 on MMLU Pro, all of which are first place ranks among Open Weight models.
• Qwen 3.5 Plus has a varied performance on our private benchmarks, ranking #30 on both the Mortgage Tax and SAGE benchmarks, but ranking #8 (and #1 among Open Weight models) on Finance Agent v1.1.
• It loses out on performance due to having a highly sensitive content filter and havng issues following a strict output format. This leads to low performance on the CaseLaw (v2) and MMMU benchmarks.

Anthropic’s latest model, Claude Sonnet 4.6, debuts at #1 on our Vals Index and #1 on the Vals Multimodal Index, evaluated across 17 benchmarks. Key takeaways:

• Claude Sonnet 4.6 takes first place on both Finance Agent (63.3%) and Tax Eval v2 (77.1%), demonstrating strong domain expertise in finance and tax.
• It takes first place on Terminal Bench 2 (59.55%), beating out Claude Opus 4.5 (Thinking) and Claude Opus 4.6 (Thinking). It is also #3 on SWE-bench (76.2%) and #2 on ProofBench (45.0%), showing competitive performance on agentic coding and formal mathematics.
• On knowledge benchmarks, Sonnet 4.6 scores 92.3% on AIME, 87.3% on MMLU Pro (#5), and 85.6% on GPQA.
• The model finishes top 10 on 13 of 17 benchmarks, with particularly strong results across finance and legal domains including #6 on CaseLaw and #7 on Mortgage Tax.