Dataset and Context
TaxEval v2 evaluates models’ abilities to answer hard tax-related questions. This version focuses on both answer correctness and structured reasoning capabilities. This dataset was created in collaboration with financial and tax experts, who have both created and double-checked all questions and answers.
Some key features:
- 1,500+ total questions across validation and test sets
- A balanced distribution of topics and question types
- Comprehensive evaluation of both answers and reasoning steps
The benchmark consists of three main components:
- Public Validation Set: 20 samples, available upon request (contact contact@vals.ai).
- Private Validation Set: 300 samples for model evaluation, available for purchase to evaluate and improve models.
- Test Set: 1,223 samples. These samples are never shared.
The benchmark is composed of two tasks (each task uses the same questions, but they are evaluated differently):
- Answer Correctness: The factual correctness of the answer, as compared to a ground truth.
- Stepwise Reasoning: The quality and structure of the reasoning process as compared to the reasoning process of human experts. This ensures models not only provide correct answers but also demonstrate a clear thinking process.
The benchmark includes a diverse range of question types:
- Application and Compliance (18.3%)
- Comparative Analysis (16.2%)
- Numerical Reasoning (16.7%)
- Problem Solving and Critical Thinking (16.5%)
- Semantic Analysis (18.0%)
- Updates and Current Affairs (15.9%)
Each category is carefully balanced between the private validation and test sets to ensure representative sampling. The evaluation process uses Claude Sonnet 4.5 (Nonthinking) as judge for both answer correctness and stepwise reasoning assessment.
Results
Model Output Examples
The hardest questions for the models typically require complex, multi-step calculations, reasoning on what laws and numbers to use given the information, or accessing more recent data.
Below is an example of a question where one model arrives at the correct answer ($18,280) while the others perform the calculations incorrectly, despite all models using a significant amount of reasoning tokens.
Michael, a married taxpayer filing jointly, has an adjusted gross income (AGI) of $300,000 for 2023, excluding any investment income. During the year, he received $20,000 in interest from corporate bonds and $15,000 in interest from municipal bonds issued by his state of residence. In addition, he sold a collectible artwork for $100,000 that he had purchased 5 years ago for $60,000. Calculate Michael's tax liability related to his investment income, including any applicable taxes on capital gains and considering the Net Investment Income Tax (NIIT).
CORRECT
Changelog
This benchmark was updated on 11/17/25 to switch the LLM-as-judge from Claude 3.5 Sonnet, which had been deprecated and is no longer usable, to Claude 4.5 Sonnet.