ModelRefs / HellaSwag — AI Glossary
HellaSwag — AI Glossary
A commonsense NLI benchmark requiring models to choose the most plausible continuation for a partially described activity.
Overview
HellaSwag (Zellers et al. 2019) uses adversarial filtering to generate hard negatives that fool BERT-era models but are easy for humans (95%). Tests grounded commonsense reasoning about physical and social situations. GPT-4 achieves ~95%; smaller 7B models score 80–88%.
Reference details
| Topic | evaluation |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: What the four options actually are
The model gets a short context — someone begins a familiar physical activity — and four possible continuations, of which one is the real caption from the source and three were machine-generated. The wrong options were not written to be obviously wrong: they were kept precisely because a discriminator model of the period was fooled by them while humans were not. That construction is the whole design, and it is also the caveat. Progress on the set partly measures not being fooled by a 2019-era generator, which is a narrower claim than commonsense reasoning.
Commonly confused with
HellaSwag is multiple choice scored by likelihood over fixed options, so it tests discrimination, not generation. A model that picks the right ending cannot necessarily write one. Read it beside a generative benchmark rather than as a summary of reasoning ability.
When to use it
Reach for it when:
- Regression checks after quantization, pruning or distillation, where a drop signals real damage
- Smoke-testing a small or fine-tuned model against a widely reported baseline
- Cheap, fast, deterministic scoring that needs no judge model
Reach for something else when:
- Separating frontier models, which cluster near the human ceiling
- Any claim about agentic or long-form ability — the format cannot reach it
- Comparing published figures without checking the prompt format and normalisation used
Primary source
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to HellaSwag — AI Glossary.
Frequently asked questions
What is HellaSwag?
A commonsense NLI benchmark requiring models to choose the most plausible continuation for a partially described activity.
What concepts are related to HellaSwag?
Closely related concepts include arc challenge, winogrande, truthfulqa.