Motivation
As AI capabilities rapidly advance, understanding their potential to transform economic sectors has become increasingly critical for organizations making deployment decisions. Unlike existing aggregated metrics that treat all capabilities equally, the Vals Index is designed to reflect the potential economic impact of AI models on the U.S. economy. We accomplish this by computing a weighted average of model performance across key sectors, where the weights correspond to each sector’s contribution to the U.S. economy in trillions of dollars.
Vals AI has developed a comprehensive suite of benchmarks measuring AI models’ ability to perform real-world tasks across finance, software engineering, and education. These benchmarks were designed to evaluate practical performance on actual professional workflows, making them well-suited for assessing economic impact. The Vals Multimodal Index leverages this existing work to provide a high-signal measure that accounts for the real-world tradeoffs between capability, latency, and cost that practitioners face when deploying AI systems.
Results
Key Takeaways
Kimi K3 is the only open-weight model in the leading tier of this composite of hard, image-inclusive economic tasks. Claude Fable 5 leads at 74.15%, with Claude Opus 5 at 73.90% and Kimi K3 at 73.42%, all within a point of one another; GPT-5.6 Sol follows at 72.64%.
Methodology
Benchmark Selection, Economic Weighting, and Formula
The Vals Index aggregates performance across three major economic sectors, weighted by their approximate contribution to U.S. GDP. Market size estimations were computed based on data from the Federal Reserve Economic Data (FRED) and the Bureau of Labor Statistics. While this represents a vast oversimplification of how AI might impact the economy, it provides a useful proxy for measuring the potential economic significance of model capabilities:
Finance (weight: 2.0): ~$2T contribution to U.S. GDP
- CorpFin: Corporate finance document analysis
- Finance Agent v2: Multi-step financial reasoning tasks
- MortgageTax: Mortgage and tax document analysis
Coding (weight: 1.4): ~$1.4T contribution to U.S. GDP
- SWE-bench Verified: Real-world software engineering tasks
- Terminal-Bench 2.1: Command-line interface problem solving
- Vibe Code Bench: End-to-end app-building tasks
Education (weight: 0.3): ~$270B contribution to U.S. GDP
- SAGE: Grading handwritten student work in mathematics
These weights combine in the following formula:
Coding = 0.25 * SWE_Bench + 0.25 * TBench + 0.5 * VibeCodeBench
Vals_Multimodal_Index = (2.0 * AVG(CorpFin, FinanceAgent, MortgageTax) + 1.4 * Coding + 0.3 * SAGE) / 3.7
The denominator (3.7) normalizes the index to a 0-100 scale, where the score represents the weighted average performance across sectors proportional to their economic contribution.
Subset Selection Process
To enable efficient and cost-effective evaluation while maintaining strong correlation with full benchmark performance, we developed representative subsets for three benchmarks:
Selection Methodology: To balance evaluation efficiency with accuracy, we created representative subsets for select benchmarks using a sampling process that maximizes correlation with full benchmark scores. We validated this approach using holdout models to ensure that subset performance reliably predicts full benchmark results.
Benchmark-Specific Subsets:
- SWE-bench Verified: 33 randomly sampled instances from each difficulty level (categorized by solution time: <15min, 15min-1hr, 1-4hr, >4hr), plus all 3 instances from the hardest category
- CorpFin: 3 randomly selected questions per unique document from the original test set
- Finance Agent v2: 13-model multimodal index subset evaluated with three runs per model
- Vibe Code Bench: 22 representative app-building tasks selected to cover a range of UI, data, and workflow patterns
Full Benchmarks:
This methodology ensures the Multimodal Vals Index provides a rapid, cost-effective evaluation framework while maintaining the predictive validity needed for reliable model comparison.
Updates
5/27/2026
- Updated the coding bucket to Terminal-Bench 2.1. The Vals Multimodal Index now evaluates coding performance with Terminal-Bench 2.1 while keeping the same 0.25 coding weight.