Key Takeaways
- Unlike the saturated knowledge benchmarks, IOI still sharply separates models: Claude Opus 5 leads at 91.67% overall, but scores fall off steeply just below the top few, so competitive-programming ability remains a real differentiator.
- Recency shows a mixed picture: Claude Opus 5 and Claude Fable 5 both achieve 100% on 2024 but drop to 83.33% and 44.50% respectively on 2025. Yet across the full cohort, mean accuracy is nearly flat between years (22.10% vs 22.29%), with nearly equal numbers of models improving or declining, suggesting the 2025 set is not uniformly harder — rather, it exposes specific weaknesses in current top models.
Why IOI?
Recently, top LLM labs like OpenAI and Google reported that their models achieved gold medals on the International Mathematical Olympiad (IMO). However, advanced models are starting to saturate IMO, meaning it may no longer effectively differentiate between the capabilities of top-performing models. Reports also suggest the evaluation process faced coordination challenges, with AI companies seeking expedited validation mid-competition that may not reflect standard IMO assessment procedures.
The International Olympiad in Informatics (IOI) offers several advantages as an LLM benchmark. Unlike the IMO, the IOI is not yet saturated, providing clear differentiation between model capabilities. The competition features standardized and automated grading, ensuring objective evaluation without subjective scoring. Additionally, the IOI has real-world relevance as it tests C++ programming skills that are directly applicable to software development.
Benchmark Design
We designed our benchmark to imitate competition conditions as closely as possible.
Agent Harness
We adapted our open-source agent harness from our Finance Agent Benchmark by providing it with access to the following tools:
- a
c++(v20) execution environment, which executes arbitrary code - a submission tool used for grading, which executes submitted code and returns a score out of 100 total possible points
These tools, particularly the submission tool, were based on the testing environment available to human contestants.
Our implementation is open source in the IOI agent GitHub repository.
Scoring
We further designed our submission tool to match the grading process during the olympiad. Agents get up to 50 submissions, each of which are graded on a variety of subtasks. A competitor receives credit for a given subtask if any submission passes all tests on a given subtask. In particular, this means the final score can be higher than any individual submission, since credit for separate subtasks are combined by the grading system.
Results
Claude Opus 5 leads with 91.67% accuracy, followed by GPT-5.6 Sol at 86.67% and Qwen 3.8 Max at 73.00%. GPT-5.6 Luna (72.92%) and Claude Fable 5 (72.25%) complete the top five. Performance remains uneven across years and models, with a steep drop after the top two results.
We chose to evaluate on both 2024 and 2025 to check for data contamination, which we suspect to explain the performance decrease between those years on LiveCodeBench. Across the current model cohort, average performance is essentially flat between the two years, with 2025 slightly higher.