ModelRefs / Evaluation Benchmark — AI Glossary
Evaluation Benchmark — AI Glossary
A standardized dataset and scoring protocol for measuring model capability on a specific task or domain. A benchmark is not an evaluation of your system.
Overview
Benchmarks (MMLU, GPQA, SWE-Bench, HumanEval) enable cross-model comparison. Risks: contamination (test data in training), saturation (frontier models max out), and gameability (models tuned to specific benchmarks). Production evals on private data remain essential alongside public benchmarks.
Reference details
| Topic | evaluation |
|---|---|
| Also known as | benchmark, eval benchmark |
| Last reviewed | 2026-06-24 |
Related terms
Example: Saturation and contamination, concretely
A benchmark where frontier models all score 88-91% has stopped discriminating — the remaining gap is mostly noise and label error. Separately, if the test set predates the model's training cutoff and is public, some of it is plausibly in the training data. Both problems make a headline number look more meaningful than it is.
Commonly confused with
A benchmark is not an evaluation of your system. It measures a model on a shared public task under fixed conditions. Your eval measures your pipeline on your data against your criteria — and a model that leads a leaderboard can still lose on the one that governs your release.
When to use it
Reach for it when:
- Shortlisting models before committing to your own evaluation
- Tracking capability trends across releases
- You need a shared reference others recognise
Reach for something else when:
- As a release gate — it does not measure your task
- When the benchmark is saturated or plausibly contaminated
- Comparing scores from different harnesses or few-shot settings
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Evaluation Benchmark — AI Glossary.
Frequently asked questions
What is Evaluation Benchmark?
A standardized dataset and scoring protocol for measuring model capability on a specific task or domain.
Is Evaluation Benchmark the same as benchmark?
Yes — benchmark, eval benchmark are common aliases for Evaluation Benchmark.
What concepts are related to Evaluation Benchmark?
Closely related concepts include eval, llm as judge, mmlu, swe bench.