Most frontier models let you choose how hard they think when you ask a question, but the higher settings increase the cost per run. By default, we generally run models on their highest setting. In this post, we measure how performance and cost change across reasoning levels. At their lowest setting, four of the seven models kept at least 80% of their Vals Index score for half the cost or less. Claude Opus 5.5, for example, kept 80% of its score for 16% of the cost.
We tested the top model from each of seven AI labs on the eight benchmarks in the Vals Index. Each model ran at three settings: its lowest reasoning setting, a middle one, and the setting our leaderboard runs use, usually its highest. In this post, we refer to the last one as the leaderboard setting. The chart below shows all three settings on the Vals Index, starting with four models, and the model legend adds the rest. The same chart is on the Vals Index page and on each of its benchmark pages.
Key Takeaways
- Lowering reasoning effort reduces cost much more than it reduces score. At their lowest setting, four of the seven models kept at least 80% of their score for half the cost or less.
- In 22 of the 56 combinations of a model and a benchmark, a lower setting scored as high as the leaderboard setting at lower cost. In those cases, the leaderboard setting cost more without scoring higher.
- Even with perfect hindsight, picking the highest-scoring setting for each task adds only 6.0 points. It also costs more than using the lowest setting everywhere, at 64% of the leaderboard cost against 29%.
Lowering reasoning effort reduces cost much more than score
The dashed line marks the leaderboard setting. A bar at 100% has the same score or cost as that setting.
At the lowest setting, four of the seven models kept at least 80% of their Vals Index score for half the cost or less. Claude Opus 5.5 kept 80% of its score at 16% of the cost, and Grok 4.7 kept 86% at 16%. Gemini 4 Argon kept 99% of its score at 84% of the cost. At the middle setting, Gemini 4 Argon scored slightly above its leaderboard setting at 78% of the cost, which is cheaper than its own lowest setting.
In 22 of 56 cases, a lower setting scored as high at lower cost
In 22 of the 56 model-and-benchmark pairs, a cheaper setting scored as high as the leaderboard setting. Of the cheaper settings we tested, none scored significantly higher than the leaderboard setting, and 33 scored significantly lower. On the Vals Index as a whole, no model matched its leaderboard score at its lowest setting.
On the Vals Index, 0 of 7 models scored as high as the leaderboard setting, at lower cost, with their lowest setting. Across both settings and all benchmarks, a lower setting scored as high at lower cost in 22 cases, none significantly different from the leaderboard setting.
Picking the highest-scoring setting for each task adds only 6.0 points
With hindsight, the models gained a median 6.0 points over their leaderboard setting, at a median 64% of the cost. A real system must choose before it sees the results, so its gain would be smaller.
Some tasks need more reasoning than others, so a system that picked a setting for each task could beat any single setting. To find the most any such system could gain, we looked back at our results and, for each task, chose the cheapest setting that got the best score.
This chart uses each task's results to pick its highest-scoring setting. A system that chooses before seeing those results would score lower. The score axis starts at 40.
More reasoning helped some benchmarks far more than others
The score increase from the lowest setting to the highest varied by benchmark. Legal Research had a median increase of 19.2 points. EMB, HLAB and Finance Agent 2.1 each increased by less than 6 points.
Change in score points, highest effort minus lowest
Methodology
We ran DeepSeek V4.1 Flash, Grok 4.7, Muse Spark 1.3, GPT-6 Astra, Claude Opus 5.5, GLM 5.3 and Gemini 4 Argon, each at its lowest reasoning setting, a middle setting and its leaderboard setting. The leaderboard setting is max for DeepSeek, Astra, Opus and GLM, and xhigh for Grok and Muse. Gemini’s leaderboard runs set no effort level, so we used its high setting. We ran them on the eight benchmarks that make up the Vals Index: Finance Agent 2.1, EMB 1.1, Terminal-Bench 4, Vibe Code Bench, Code Migration 1.1, Legal Research, HLAB 1.1 and Tax Agent Bench.
Every setting ran on the same fixed subset of each benchmark, about half the tasks. We chose each subset so that it ranks the models in the same order as the full benchmark. For most models, the leaderboard-setting score is the run already on our leaderboard, rescored on that subset. Where no such run existed, including all of Gemini’s, we ran the leaderboard setting fresh.
The Vals Index is a single measure of AI’s potential economic impact. It measures agentic model performance across finance, coding, legal and tax tasks, weighted by each sector’s share of U.S. GDP. A model appears in the Index only when all eight benchmarks have a score. Tax Agent Bench is scored all-pass, as in its main table. Cost per test is at list price. Vals Index cost is the plain mean of the eight benchmarks’ costs, with Code Migration estimated from tokens. Each comparison uses a paired bootstrap test, which resamples the same tasks for both settings to judge whether a score difference is likely real. We ran every comparison in a group before checking which setting scored higher, then adjusted the p-values across each group with the Benjamini–Hochberg method. A result counts as significant when its adjusted p-value is below 0.05. To pick each task’s highest-scoring setting, we take the cheapest setting with the highest score on that task, then compare plain averages over the eight benchmarks.
Limitations
- Scores are on half the tasks, so each is noisier than the published number, and most benchmarks have one run per setting.
- No significant difference does not mean the scores are equal. With about half the tasks per benchmark, our test can miss a small real difference between two settings.
- Most leaderboard-setting runs are from earlier months and the other settings from October, so provider-side changes in between could affect the comparison.
- We picked each task’s highest-scoring setting from that task’s own results, so the 6.0-point gain is an upper bound on what a real system could achieve.