TaxEval

Key Takeaways

• Model choice barely matters at the top: the leaders top out around 80% and the top ten sit within about five points of each other, so picking among frontier models buys little accuracy here.
• The benchmark’s two evaluation dimensions expose separate ceilings: Muse Spark 1.2 scores 92.72% on stepwise reasoning but only 68.03% on answer correctness. The nearly 25-point gap shows that despite strong reasoning demonstration, the frontier challenge is accurate final computation, not reasoning structure.

Dataset and Context

TaxEval v2 evaluates models’ abilities to answer hard tax-related questions. This version focuses on both answer correctness and structured reasoning capabilities. This dataset was created in collaboration with financial and tax experts, who have both created and double-checked all questions and answers.

Some key features:

• 1,500+ total questions across validation and test sets
• A balanced distribution of topics and question types
• Comprehensive evaluation of both answers and reasoning steps

The benchmark consists of three main components:

• Public Validation Set: 20 samples, available upon request (contact contact@vals.ai).
• Private Validation Set: 300 samples for model evaluation, available for purchase to evaluate and improve models.
• Test Set: 1,223 samples. These samples are never shared.

The benchmark is composed of two tasks (each task uses the same questions, but they are evaluated differently):

1. Answer Correctness: The factual correctness of the answer, as compared to a ground truth.
2. Stepwise Reasoning: The quality and structure of the reasoning process as compared to the reasoning process of human experts. This ensures models not only provide correct answers but also demonstrate a clear thinking process.

The benchmark includes a diverse range of question types:

• Application and Compliance (18.3%)
• Comparative Analysis (16.2%)
• Numerical Reasoning (16.7%)
• Problem Solving and Critical Thinking (16.5%)
• Semantic Analysis (18.0%)
• Updates and Current Affairs (15.9%)

Each category is carefully balanced between the private validation and test sets to ensure representative sampling. The evaluation process uses Claude Sonnet 4.5 (Nonthinking) as judge for both answer correctness and stepwise reasoning assessment.

Results

Model Output Examples

The hardest questions for the models typically require complex, multi-step calculations, reasoning on what laws and numbers to use given the information, or accessing more recent data.

Below is an example of a question where one model arrives at the correct answer ($18,280) while the others perform the calculations incorrectly, despite all models using a significant amount of reasoning tokens.

Changelog

This benchmark was updated on 11/17/25 to switch the LLM-as-judge from Claude 3.5 Sonnet, which had been deprecated and is no longer usable, to Claude 4.5 Sonnet.