MMLU (Massive Multitask Language Understanding) — AI Glossary

Reference details

Related terms

• Evaluation Benchmark
• GPQA Diamond

Example: Why the headline number stopped discriminating

Four-choice questions mean random guessing scores about 25%. When frontier models cluster in the high eighties, the remaining spread is a few points sitting on top of known label errors and possible contamination. The benchmark still separates weak models from strong ones; it no longer separates strong from strong.

random baseline = 1 / 4 = 25%
frontier cluster ~ high 80s  -> a few points of usable spread

Commonly confused with

MMLU measures recalled knowledge under multiple choice, not reasoning. The format lets a model succeed by elimination without following an argument — which is exactly what MMLU-Pro was built to reduce by adding options and harder distractors.

When to use it

Reach for it when:

• Coarse capability screening across a wide subject range
• Tracking generational improvement where the gap is large
• You need a benchmark others will recognise

Reach for something else when:

• Separating frontier models, where it has saturated
• As evidence of reasoning ability
• Comparing numbers from different harnesses or few-shot settings

Primary source

• Hendrycks et al., Measuring Massive Multitask Language Understanding (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MMLU (Massive Multitask Language Understanding) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path