May 5, 2025
Google's Gemini 2.5 Flash (Nonthinking) evaluated on most benchmarks
We just evaluated Gemini 2.5 Flash Preview (Nonthinking) on most benchmarks.
- Gemini 2.5 Flash Preview (Nonthinking) is a lightweight alternative to Google’s flagship model, Gemini 2.5 Pro Exp. Gemini 2.5 Flash Preview (Nonthinking) runs at a fraction of the cost and latency, rendering it a more accessible option.
- Like Claude 3.7 Sonnet (Nonthinking), Gemini 2.5 Flash Preview (Nonthinking) is a hybrid reasoning model, meaning it can adaptively choose how much to think before responding.
- Gemini 2.5 Flash Preview (Nonthinking) excels on LegalBench, coming second only to the flagship Gemini 2.5 Pro Exp (and outperforming its own thinking variant, Gemini 2.5 Flash Preview (Thinking), by 1%).
- We consistently had difficulty with Google’s API during evaluation, which prevented us from reporting full results. We’re working with a representative from the Gemini team to resolve those issues.