We evaluated Claude Haiku 4.5 (Thinking) and found strong performance:

• The model places 3rd on our Vals Index, demonstrating well-rounded capabilities across diverse tasks.
• On Terminal Bench, Haiku 4.5 achieves 3rd place, showing particular strength on coding tasks.
• While it performs well on certain coding benchmarks, the model achieves middle-of-the-pack performance on most other benchmarks.
• The model struggles significantly on our proprietary CaseLaw benchmark and the public MedQA, GPQA, MMLU Pro, MMMU, and LiveCodeBench benchmarks.
• Compared to Claude Sonnet 4.5 (Thinking) , Haiku trades some performance for significantly faster speed and lower cost - see our model comparison for details.

Overall, Haiku 4.5 sits firmly on the Pareto frontier.

Our new Student Assessment with Generative Evaluation (SAGE) benchmark evaluates the ability of Language Models to grade handwritten student work at the undergraduate level.

• No models score higher than 50%; by contrast, top models consistently achieve >90% accuracy on mathematical benchmarks like AIME.
• Gemini 2.5 Flash (7/17) (Nonthinking) takes first place with 44.8% - outperforming Gemini 2.5 Pro by 3%!
• Current results show significant variation in performance by model provider, with less variation in particular model. With the exception of the two smallest models we tested ( Gemini 2.5 Flash Lite (Nonthinking) and GPT 5 Nano ), all OpenAI and Google models do better than all Anthropic models, which in turn do better than Grok 4 .

We find significant room for improvement in models’ capabilities here, and are excited for the SAGE benchmark to enable continued model development for education!

We evaluated Magistral Medium 1.2 (09/2025) and Magistral Small 1.2 (09/2025) - and found that both models perform decently for their size, especially on coding tasks. However, the models also struggled on many benchmarks.

• Magistral Medium performs well on academic and coding benchmarks, placing in the top 20 on LiveCodeBench and AIME. However, the model struggles on our proprietary benchmarks, particularly MortgageTax and CaseLaw.
• Surprisingly, Magistral Small tends to do better on finance and academic benchmarks, most notably outperforming Medium on MortgageTax (+8.8%). The model also does well on LiveCodeBench and AIME. However, Small struggled on our proprietary CorpFin and CaseLaw benchmarks, along with GPQA and MMLU Pro.
• A large chunk of the performance loss was the result of models not outputting results in the format that was required.

The Medium model is priced at $2 / $5, and the Small at $0.5 / $1.5. The Small model has open weights, whereas the Medium model is only available via API.

We ran the recently-released Claude Sonnet 4.5 (Thinking) our benchmarks, and found very strong performance:

• On Finance Agent, it beats the previous state-of-the-art by five percentage points.
• It also takes the #1 spot on SWE Bench and Terminal Bench, beating out GPT 5 Codex .
• It is in the top 10 models on the majority of our benchmarks, and also showed better performance than Claude Sonnet 4 (Thinking) on almost all benchmarks.
• It has a 1-million token context window, when the flag “context-1m-2025-08-07” is enabled

Overall, this model is extremely capable, at the same mid-range price point as its predecessor.