We evaluated Gemini 3.1 Flash Lite Preview across our full benchmark suite. The model is Google’s fast and cost-efficient offering.

• The model does well on two of our multimodal benchmarks, SAGE (ranked 5th, with 49.5% accuracy) and Mortgage Tax (ranked 7th, 67.8% accuracy).
• It ranks 14th on MMLU Pro with 86.2% accuracy.
• On coding tasks the model still has room for improvement—it ranks 29th on both Live Code Bench and Terminal Bench 2, and 28th on SWE-bench. It scores 0% on Vibe Code Bench.
• Overall, it places 15th/20 on the Vals Multimodal Index and 22nd/31 on the Vals Index.
• The cost savings compared to other models in the Gemini 3 series or Gemini 2.5 are dramatic: roughly 5–20x cheaper per test across benchmarks, while maintaining respectable accuracy. For example, on Finance Agent it costs $0.072 per test vs $0.370 for Gemini 3 Flash, while performing comparably.

Our results show that the model does not perform as well as other models in the Gemini 3 series. However, it is fast and quite cost-efficient relative to those models, making it a good choice for applications that demand scale, speed, or cost-efficiency.

Evaluations were run with a temperature of 1.0 and a “high” thinking level, via the official Google API.

We’ve released updated results for the CaseLaw (v2) benchmark, where we’ve performed additional runs of each model to produce more reliable results.

Averaging over multiple runs ensures that the shown rankings are a reflection of the models’ true performances, rather than a one-off result.

OpenAI’s GPT 5.1 takes first place with a score of 73.42%.

We evaluated Qwen 3.5 Flash on the Vals Index. Here are the key takeaways:

• Qwen 3.5 Flash places #22 overall on Vals Index (49.6% accuracy), and #7 among open-weight models.
• The model is strongest on financial tasks: it reaches 64.2% on our Corp Fin (v2) subset and 55.9% on Case Law (v2).
• It scores 54.9% on SWE-bench but struggles on Terminal-Bench 2.0 (24.7%) and Finance Agent (46.7%).
• At ~$0.08 per test on the index, it offers strong cost-effectiveness for its tier.

We’re releasing Vibe Code Bench v1.1.

This update makes several improvements to both the data and the evaluator for Vibe Code Bench.

The v1.1 update improves data quality by standardizing authentication modes, using explicit pre-seeded emails, and clarifying instructions (on a small subset of tests).

On the evaluation side, the update enhances the browser-use evaluator with new tools for interacting with complex HTML elements and better handling for “waiting”, as well as providing it with an arsenal of files (image and others) to use during testing.