ModelRefs / GAIA — AI Glossary

GAIA — AI Glossary

A benchmark of real-world assistant tasks requiring multi-step web research, tool use, and multi-modal reasoning at human difficulty.

Overview

GAIA (Mialon et al. 2023, Meta) presents 466 tasks that a human can solve in under 2 minutes but current AI agents struggle with: navigating multi-step web research, parsing PDFs, using code tools, and reasoning over heterogeneous sources. Human accuracy ~92%; GPT-4 + tools at launch ~15–30%. The defining agent benchmark for 2024.

Reference details

Topic	evaluation
Last reviewed	2026-06-24

Example: Graded end to end, so one broken step is a zero

Tasks are things a capable person finishes in a couple of minutes but which need several steps: find a page, read a table in a PDF, do a small calculation, combine it with something else. Grading is on the final answer only. That means a run which does four steps perfectly and misreads the fifth scores exactly the same as one that never started — and it means the benchmark measures the whole system, since a browsing tool that fails on a redirect will sink an otherwise capable model. Useful as a realistic difficulty signal; misleading if read as a property of the model alone.

Commonly confused with

This is an assistant-task benchmark, not a reasoning or knowledge benchmark. Difficulty comes from orchestration — many steps, heterogeneous sources, tools that fail — rather than from any single step being hard. That makes it sensitive to scaffold quality in the same way agentic coding benchmarks are, and results should name the tool set used.

When to use it

Reach for it when:

Evaluating a complete agent stack, tools included, rather than a bare model
Comparing scaffolds and tool designs against a fixed model
As a realistic difficulty check for research-style assistant workloads

Reach for something else when:

Comparing models without holding the tool set and browsing environment constant
Attributing failure to reasoning without reading which step actually broke
As a general capability score — the task distribution is deliberately specific

Primary source

Mialon et al., GAIA: a benchmark for General AI Assistants (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GAIA — AI Glossary.

Frequently asked questions

What is GAIA?