Results
According to the original MMMU research, human expert performance ranges from 76.2% for the worst-performing experts to 88.6% for the best-performing experts. Our evaluation shows that the leading AI models have now slightly surpassed the upper bound of reported human expert performance.
Dataset and Context
The Multimodal Multi-task Benchmark (MMMU) follows a similar methodology to its predecessor, MMLU, but the multiple-choice questions asked include both text and images. MMMU Pro encompasses over 1,000 high-quality tasks spanning 30 subjects in 6 major disciplines:
- Arts & Design
- Business
- Science
- Health & Medicine
- Humanities & Social Sciences
- Tech & Engineering
We based this benchmark on the standard 4-option multiple-choice format containing approximately 1,700 questions from the official Hugging Face dataset. The benchmark focuses specifically on how well models can process and reason about problems where images are interleaved with text, requiring sophisticated visual understanding and cross-modal reasoning capabilities.
MMMU Pro is particularly valuable because it tests the models’ abilities to solve graduate-level questions where visual information is critical to finding the correct answer.
Methodology
We adhered closely to the official MMMU Pro evaluation protocol with the following implementation details:
- Prompt Structure: We used the chain-of-thought prompt from the original MMMU repository. Each question-answer set followed this format:
Which of the following best explains the overall trend shown in the <image>?
A. Migrations to areas of Central Asia for resettlement
B. The spread of pathogens across the Silk Road
C. Invasions by Mongol tribes
D. Large-scale famine due to crop failures
Answer the preceding multiple choice question. The last line of your response should be of the following format: 'Answer: $LETTER' (without quotes) where LETTER is one of options. Think step by step before answering.
-
Image Processing: All
<image>tags were replaced with actual image bytes during model inference, replicating the original methodology exactly. -
Standardization: We established a consistent baseline by using a default configuration of 8192 maximum output tokens for all models to ensure that outputs were not truncated. All models were ran with a temperature of 0.
-
Parsing Adaptation: We modified the answer extraction regex to handle markdown output from some models, ensuring reliable parsing across all responses.
-
Statistical Validity: This was a pass@1 evaluation (one attempt per question), with the large dataset size (1,700 questions). We found a standard deviation approximately 1% (calculated using the methodology from Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations)
-
Thinking Models Configuration: For models with “thinking” capabilities (Claude 3.7 Sonnet Thinking, o1, etc.), we set the maximum output token limit to 16,384 to accommodate extended reasoning.