ModelRefs / MMLU-Pro Leaderboard — AI Model Scores

MMLU-Pro Leaderboard — AI Model Scores

Harder MMLU successor with 10-option questions and reduced contamination. Current leaders, methodology, and citation sources for MMLU-Pro.

Overview

Harder MMLU successor with 10-option questions and reduced contamination.

How it is measured: 5-shot CoT across 14 disciplines; 12k questions, 10 answer choices.

What this benchmark measures

  • multi-discipline reasoning
  • robust multiple-choice selection

Relevant to:

  • hard reasoning evaluation portfolios
  • broad discipline screening

Failure modes it exercises:

  • reasoning errors under larger distractor sets

Method and its limits

Five-shot chain-of-thought evaluation across 14 disciplines with ten answer choices.

  • Discipline-level performance can be obscured by aggregate accuracy.
  • Chain-of-thought and harness settings can affect comparability.

Dataset

Dataset
MMLU-Pro
Type
academic multiple-choice questions with ten options
Freshness
aging

The registered methodology states over 12,000 questions across 14 disciplines and ten answer choices. The 2024-11-06 arXiv revision anchors the reviewed methodology, but it is not a current-world knowledge cutoff.

How to read this score

Accuracy supports matched-harness comparison, but should be decomposed by discipline and paired with real-task evaluation.

Similarity to real tasks: Low — Ten-option questions increase difficulty, but remain controlled academic multiple-choice tasks.

Data contamination risk: Medium — MMLU-Pro was designed to reduce trivial/noisy MMLU artifacts and is newer than MMLU, but the data and evaluation code are public; model-specific training overlap still requires run-level evidence.

Benchmark gaming risk: Medium — The paper reports lower prompt sensitivity than MMLU, but chain-of-thought use, prompt style, dataset revision, and benchmark-specific tuning still affect comparability.

What you still need to test yourself

  • Test open-ended, current, domain-specific, and production-format tasks.
  • Pin the dataset revision and evaluate prompt sensitivity, calibration, safety, cost, and latency separately.

This benchmark supports decisions about:

  • Compare broad multi-discipline reasoning under a matched MMLU-Pro harness.
  • Use discipline-level results to identify where deeper evaluation is needed.

Limitations

  • Does not test deployment constraints, safety, privacy, tool use, or open-ended production performance.
  • A higher aggregate score does not establish universal model superiority.
  • Every score needs a canonical run record with exact model and dataset versions, prompt/harness settings, source scope, and limitations; no model-specific contamination conclusion is inferred.

Sources re-reviewed 2026-07-11. MMLU-Pro is treated as an aging 2024 benchmark with a reviewed v6 paper revision; each score must still identify the dataset and harness revision used.

Sources

How this benchmark is scored

Categoryreasoning
Maximum score100 % accuracy
DirectionHigher is better

Primary source: https://arxiv.org/abs/2406.01574

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MMLU-Pro Leaderboard — AI Model Scores.