The independent evaluator of artificial intelligence. We build
the benchmarks and evaluation infrastructure that measure
whether models can do the work of lawyers, bankers, engineers,
and doctors.
Rapid progress in AI has been driven by a process researchers call hill-climbing: repeatedly defining a measurable objective, identifying shortcomings, and improving against them. Every major advance in AI, from image classification to question-answering to software development, has been measured this way. One hill at a time.
The pace of model development is outpacing the community’s ability to construct new hills. Trillions have been invested in generating intelligence, but comparatively little in measuring it. As a result, models summit old benchmarks in months. Test sets released openly are absorbed into pre-training corpora, which quietly invalidates the results built on them. And new benchmarks are often produced or run by the same companies that build the models, reported alongside cherry-picked examples and evaluation regimens tuned to score well.
AI has quickly become a trillion-dollar market without the independent measurement institutions that other markets of this scale, like finance and healthcare, require. The industry is worse off because of it.
Labs lack a credible way to demonstrate continued model progress.
Enterprises increasingly see pressure to adopt and spend on AI without means to quantify ROI.
Governments must develop their own expertise to measure frontier capabilities, cyber risk, and the pace of global competition.
We started Vals AI to solve this problem, as the independent evaluator of artificial intelligence.
What We Build
We build benchmarks that measure the ability of models to do the work of lawyers, bankers, engineers, and doctors. This takes partnering with reference institutions in each field to build a taxonomy of representative tasks, and developing new methodologies to automatically score the quality of generated work product. These benchmarks, including Finance Agent, have rapidly become industry standards.
Scores are based on our privately held test sets to preserve the integrity and signal of our results. This prevents training to the test set, unlike in open-source benchmarks.
We have built, and even open-sourced, the infrastructure we rely on to run these evaluations reproducibly and at scale across labs. This includes Valkyrie, a distributed system to run agentic benchmarks, and our model library, a free standard API to call models with.
We also produce benchmarks we believe will be broadly beneficial to the public good. For example, Public Benefits Bench, built with the Center for Civic Futures and Code for America, measures whether AI can be trusted to answer SNAP questions for the 37 million families who rely on the program.
Whether you build AI, evaluate it, or want to help shape a
benchmark in your field, we would love to hear from and work
with you.
Methodology
Our benchmarks measure the capability and reliability of AI
models and agents in realistic tasks. In contrast with
contrived exam-style benchmarks, we focus on economically
valuable and scientifically important domains—finance,
healthcare, math, coding, and more. Developed in collaboration
with domain experts, our datasets are carefully curated to be
of extremely high quality and push models to their limits.
Task Design
Our benchmarks reflect the complexity of real-world tasks, which necessitates evaluating multiple types of capabilities:
Tool-Use: How well can models call the right tools to solve problems?
Multiple Modalities: How well can models handle images, tabular data, files, and other modalities beyond text?
Reasoning: Models are increasingly trained to output reasoning before answering; do these capabilities actually improve real-world utility?
Long-Context Capabilities: Can models reason over long contexts, such as extensive legal documents or large codebases?
Long-Horizon Tasks: Can models autonomously work on tasks that take minutes, hours, or longer?
Public and Private Sets
A major problem with evaluations of AI models is test-set leakage 1. Benchmark data can contaminate training sets either directly or through synthetic data 2, undermining the validity of reported results.
Thus, we offer private benchmarking; for transparency and fairness, we offer (for most benchmarks):
Public Validation Set: A completely open dataset, to provide transparency in the types of samples we use for evaluation.
Private Validation Set: A larger, privately held dataset, which we license for companies to do their own internal validation. We provide (statistical) proof that this is correlated with our test suite.
Test Set: This dataset remains private at all times, and is the only dataset that is used for the benchmarks we publish. It is private to prevent leakage into the training dataset of foundation models.
Metrics and Evaluation
Benchmarks often report only accuracy numbers; however, it is important to consider factors such as efficiency, cost, time taken per test, failure modes, and more.
Our evaluation framework provides detailed insights into model performance through multiple metrics:
Accuracy: Evaluates the correctness of model outputs for each task and benchmark. This includes strict accuracy checks, as well as rubric-based LLM-as-a-judge accuracy metrics.
Latency: Measures the response time of models in returning a complete response.
Cost: Analyzes the operational cost of running each model from an API provider.
Additional quantitative and qualitative insights: For each benchmark, we also provide further information, including but not limited to
statistics regarding tool-use, qualitative insights about the nature of the errors, and comparisons between different models. This provides information beyond
the raw benchmark numbers, and also helps contextualize the performance of models.
This information enables us to offer a more comprehensive, holistic view of model performance, including accuracy, reliability, efficiency, and qualitative insights.
Error Bars
We report standard errors alongside benchmark scores to reflect statistical uncertainty.
Our methodology depends on how the benchmark is structured:
Single-run benchmarks
For benchmarks evaluated once, we follow standard uncertainty reporting practice, as suggested by “Adding Error Bars to Evals:
A Statistical Approach to Language Model Evaluations” by Evan Miller 3. Error bars are computed as the standard error of the mean (SEM) over instance-level scores.
In particular, let x1,…,xn be instance-level scores. Standard error of the mean (SEM), using sample standard deviation is given by:
SE=nn−11∑i=1n(xi−xˉ)2
where xˉ=n1∑i=1nxi is the mean.
These error bars capture measurement uncertainty in the benchmark itself. They do not reflect variability across prompts, seeds, deployment settings, or the stochastic nature of LLM generation.
Multiple-run benchmarks (AIME, CaseLaw)
When a benchmark includes multiple independent runs, we compute the SEM of the score of each run, estimating uncertainty over runs rather than over individual instances.
Let a1,…,aR be the average score from each of R independent runs and the average score be aˉ=R1∑r=1Rar.
Standard error over runs is then given by:
SEruns=Rσ=RR−11∑r=1R(ar−aˉ)2
Composite benchmarks
For benchmarks that combine multiple tasks, we propagate uncertainty from each component using weighted variance pooling.
Let component standard errors be SE1,…,SEK and weights w1,…,wK.
The propagated standard error is then:
SEoverall=∑k=1Kwk∑k=1Kwk2SEk2
In all cases, we use standard statistical definitions of the standard error of the mean, with sample standard deviation where applicable.
Evaluating not just models, but agentic systems and scaffolds
Since LLMs are often used as part of agentic systems with general scaffolds, and part of larger workflows or products, it is important to design
evaluations that measure capabilities of these kinds of systems. Our benchmarks test crucial aspects of this, such as tool-calling,
multi-turn flows, coding skills and computer-use.
In the future, our benchmarks will also evolve to test not only agentic systems we design, but also custom user-provided scaffolds and products.
These benchmarks ensure comprehensive evaluation of AI systems, addressing their growing utility in real-world applications and their ability to function autonomously in larger systems.