Task Design

Our benchmarks reflect the complexity of real-world tasks, which necessitates evaluating multiple types of capabilities:

• Tool-Use: How well can models call the right tools to solve problems?
• Multiple Modalities: How well can models handle images, tabular data, files, and other modalities beyond text?
• Reasoning: Models are increasingly trained to output reasoning before answering; do these capabilities actually improve real-world utility?
• Long-Context Capabilities: Can models reason over long contexts, such as extensive legal documents or large codebases?
• Long-Horizon Tasks: Can models autonomously work on tasks that take minutes, hours, or longer?
• Agentic Work: Can models drive a coding agent, a terminal, or a browser end-to-end in a real environment, from software engineering and code migration to computer-use tasks?

Public and Private Sets

A major problem with evaluations of AI models is test-set leakage ¹. Benchmark data can contaminate training sets either directly or through synthetic data ², undermining the validity of reported results. Thus, our proprietary benchmarks are evaluated on private data. For transparency and fairness, most of them come with:

• Public Validation Set: A completely open dataset, to provide transparency in the types of samples we use for evaluation.
• Private Validation Set: A larger, privately held dataset, which we license to companies for their own internal validation. We provide statistical evidence that it is correlated with our test set.
• Test Set: This dataset remains private at all times, and is the only dataset used for the proprietary benchmark results we publish. It is private to prevent leakage into the training data of foundation models.

We also run selected public benchmarks (for example, Terminal-Bench, SWE-bench, and IOI). For these, we run every model ourselves under the same harness and settings, so results are comparable across models even though the tasks are open.

Metrics and Evaluation

Benchmarks often report only accuracy numbers; however, it is important to consider factors such as efficiency, cost, time taken per test, failure modes, and more. Our evaluation framework provides detailed insights into model performance through multiple metrics:

• Accuracy: Evaluates the correctness of model outputs for each task and benchmark. This includes strict accuracy checks, as well as rubric-based LLM-as-a-judge accuracy metrics.
• Latency: Measures the time a model takes to return a complete response, or, for agentic benchmarks, the wall-clock time to complete a task.
• Cost: Analyzes the operational cost of running each model from an API provider.
• Additional quantitative and qualitative insights: For each benchmark, we also provide further information, including but not limited to statistics regarding tool-use, qualitative insights about the nature of the errors, and comparisons between different models. This provides information beyond the raw benchmark numbers, and also helps contextualize the performance of models.

This information enables us to offer a more comprehensive, holistic view of model performance, including accuracy, reliability, efficiency, and qualitative insights.

Error Bars

We report standard errors alongside benchmark scores to reflect statistical uncertainty. Our methodology depends on how the benchmark is structured:

Single-run benchmarks

For benchmarks evaluated once, we follow standard uncertainty reporting practice, as suggested by “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations” by Evan Miller ³. Error bars are computed as the standard error of the mean (SEM) over instance-level scores.

In particular, let $x_1, \dots, x_n$ be instance-level scores. Standard error of the mean (SEM), using sample standard deviation is given by:

where $\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i$ is the mean.

These error bars capture measurement uncertainty in the benchmark itself. They do not reflect variability across prompts, seeds, deployment settings, or the stochastic nature of LLM generation.

Multiple-run benchmarks (Terminal-Bench, Finance Agent v2, BioMysteryBench)

When a benchmark includes multiple independent runs of each model (currently three), we compute the SEM over the per-run scores, estimating uncertainty over runs rather than over individual instances.

Let $a_1, \dots, a_R$ be the average score from each of $R$ independent runs and the average score be $\bar{a} = \frac{1}{R} \sum_{r=1}^{R} a_r$.

Standard error over runs is then given by:

Composite benchmarks

For benchmarks that combine multiple tasks, we propagate uncertainty from each component using weighted variance pooling.

Let component standard errors be $\mathrm{SE}_1, \dots, \mathrm{SE}_K$ and weights $w_1, \dots, w_K$.

The propagated standard error is then:

In all cases, we use standard statistical definitions of the standard error of the mean, with sample standard deviation where applicable.

Evaluating models, agents, and products

LLMs are increasingly used inside agentic systems, and as part of larger workflows or products, so we evaluate at each of these levels:

• Models: Models are called through a fixed harness that we control, so that differences in scores reflect the model rather than the scaffold.
• Agents and scaffolds: Many of our benchmarks run models inside coding and terminal agents, including a model’s own native agent where one exists (for example, Claude Code or Codex), as well as custom scaffolds provided by the teams we work with.
• Products: We evaluate end-user applications built on top of models, such as legal research and drafting tools, against the same private task sets and rubrics.

These benchmarks exercise tool-calling, multi-turn flows, coding, and computer-use in real environments, measuring how AI systems perform when they have to function autonomously as part of a larger system.

References

1. https://arxiv.org/abs/2410.08385
2. https://arxiv.org/abs/2407.07565
3. https://arxiv.org/abs/2411.00640