Background
The American Invitational Mathematics Examination (AIME) is a prestigious, invite-only mathematics competition for high-school students who perform in the top 5% of the AMC 12 mathematics exam. It involves 15 questions of increasing difficulty, with the answer to every question being a single integer from 0 to 999. The median score is historically between 4 and 6 questions correct (out of the 15 possible). Two versions of the test are given every year (thirty questions total). You can view the questions from previous years on the AIME website
This examination serves as a crucial gateway for students aiming to qualify for the USA Mathematical Olympiad (USAMO). In general, the test is extremely challenging, and covers a wide range of mathematical topics, including algebra, geometry, and number theory.
The top models now achieve near-perfect accuracy on this benchmark, although performance can vary significantly between the 2024 and 2025 question sets.
Methodology
For this benchmark, we used the thirty questions from the 2024 and 2025 versions of the test (sixty questions total), modelling our approach after the repository from the GAIR NLP Lab.
To minimize parsing errors, we instructed the models with the following prompt template.
Please reason step by step, and put your final answer within \boxed{}
{Question}
The answer was then extracted from the boxed section and compared to the ground truth.
Although a few questions included an image or diagram, all of the information needed to solve the problem was present in the question text, so we did not include these images.
Reducing variance
Given the very low size of this benchmark, we ran each model 8 times on both AIME 2024 and AIME 2025 to reduce variance. We averaged the pass@1 performance across all runs for each model.