ModelRefs / HumanEval+ Leaderboard — AI Model Scores
HumanEval+ Leaderboard — AI Model Scores
Function-level code synthesis with extended hidden tests. Current leaders, methodology, and citation sources for HumanEval+.
Overview
Function-level code synthesis with extended hidden tests.
How it is measured: pass@1; HumanEval+ adds adversarial tests to reduce overfitting.
What this benchmark measures
- function-level Python code synthesis
- hidden-test correctness
Relevant to:
- bounded code-generation screening
- coding evaluation portfolios
Failure modes it exercises:
- functional errors on generated functions
Method and its limits
Pass@1 function-generation evaluation using hidden tests; the displayed HumanEval+ label requires harness-specific review.
- Function-level tasks do not represent repository-scale engineering.
- The registered source describes HumanEval; extended HumanEval+ test provenance is not fully represented by this source alone.
Dataset
- Dataset
- HumanEval / HumanEval+
- Type
- function-level Python synthesis tasks with hidden tests
- Freshness
- aging
HumanEval is a 2021 hand-written Python function benchmark. HumanEval+ extends test coverage through EvalPlus, but scores must name the exact harness and test-suite revision.
How to read this score
Pass@1 is easy to read under a fixed harness; it should not be generalized to software-engineering quality.
Similarity to real tasks: Low — HumanEval covers 164 hand-written Python function-from-docstring tasks scored with unit tests. HumanEval+ substantially expands test coverage, but both remain isolated function-synthesis evaluations rather than repository, dependency, security, integration, review, or maintenance work.
Data contamination risk: High — HumanEval is public, long-running, and widely used; HumanEval+ adds tests but does not by itself establish that a model has not seen the prompt or related solutions.
Benchmark gaming risk: High — Scores are sensitive to pass@k, sampling count, execution sandbox, test-suite revision, and benchmark-specific tuning.
What you still need to test yourself
- Test repository context, dependencies, security, maintainability, and the target language mix.
- Execute generated code in an isolated environment and require project-specific tests and review.
This benchmark supports decisions about:
- Screen bounded Python function synthesis under a fixed HumanEval or HumanEval+ harness.
- Compare functional-correctness results only when sampling, pass@k, and test-suite conditions match.
Limitations
- Does not test secure coding, architecture, repository context, or long-term maintainability.
- Cross-report scores may use different sampling and pass@k settings.
- Every score needs a canonical run record with model version, dataset/harness revision, pass@k and sampling settings, source scope, and limitations; training overlap remains unknown unless separately documented.
Sources re-reviewed 2026-07-11. HumanEval is treated as an aging 2021 benchmark; HumanEval+ results remain harness-revision-specific rather than a single timeless dataset claim.
Sources
- HumanEval evaluation harness OpenAI · accessed 2026-07-11
- Evaluating Large Language Models Trained on Code OpenAI · accessed 2026-07-11
- EvalPlus EvalPlus authors · accessed 2026-07-11
How this benchmark is scored
| Category | coding |
|---|---|
| Maximum score | 100 pass@1 |
| Direction | Higher is better |
Primary source: https://github.com/openai/human-eval
Published results
| Model | Score | Run date | Source |
|---|---|---|---|
| GPT-5 | 96.3 | 2026-05-01 | Aggregated public reports |
| Claude Opus 4 | 95.2 | 2026-05-01 | Aggregated public reports |
| GPT-5 Mini | 89 | 2026-05-01 | Aggregated public reports |
| Mistral Large 2 | 84 | 2026-05-01 | Aggregated public reports |
Each score reflects the protocol and date of its own source run. Results from different harnesses are not directly comparable.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to HumanEval+ Leaderboard — AI Model Scores.