ModelRefs / ARC Challenge — AI Glossary

ARC Challenge — AI Glossary

A grade-school science question-answering benchmark with adversarially filtered questions that retrieve-then-answer systems fail.

Overview

ARC Challenge (Clark et al. 2018) contains 1,172 questions from US 3rd–9th grade science exams that neither word-overlap nor retrieval-based models could answer correctly. Tests systematic reasoning and scientific knowledge. Frontier LLMs score >90%; 7B models typically 75–85%.

Reference details

Topic	evaluation
Last reviewed	2026-06-24

Example: Filtered by what retrieval could not solve

The challenge split was built by keeping the grade-school science questions that word-overlap and retrieval-based solvers of the time got wrong. That construction selects hard cases efficiently, and it also shapes what a score means: the set is enriched for questions where surface matching fails, which is a narrower claim than “tests reasoning”. It shares this property with other adversarially filtered benchmarks — progress partly measures not failing the way a particular era's solver failed. Useful as a floor check on a small model, weak as a discriminator between strong ones.

Commonly confused with

The challenge split is the filtered subset; an easy split of the same questions exists and is much less informative. Figures quoted without naming the split are ambiguous, and multiple-choice scoring by option likelihood also differs from asking the model to answer generatively — the two do not produce the same number.

When to use it

Reach for it when:

Regression checks on small models after quantization or fine-tuning
Cheap, deterministic scoring with no judge model in the loop
As one of several floor checks, never as a headline capability claim

Reach for something else when:

Distinguishing frontier models, which cluster at the top of the range
Any claim about generation, agentic ability or long-form reasoning
Comparing figures without the split, prompt format and normalisation stated

Primary source

Clark et al., Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to ARC Challenge — AI Glossary.

Frequently asked questions

What is ARC Challenge?