Pretraining Data — AI Glossary

Reference details

Related terms

• Common Crawl
• Data Deduplication
• Chinchilla Scaling

Example: Compute-optimal is not deployment-optimal

The Chinchilla result says that for a fixed training budget, roughly 20 tokens per parameter minimises loss — a 7B model would take about 7e9 × 20 = 140 billion tokens. Almost no deployed model follows this. Llama 3's 8B was trained on roughly 15 trillion tokens, about 1,875 tokens per parameter — nearly a hundred times the compute-optimal ratio. The reason is that Chinchilla optimises training cost, while a model serving billions of requests is dominated by inference cost. Over-training a smaller model buys a permanently cheaper model to run.

Commonly confused with

Pretraining data is not fine-tuning data and not a knowledge base. It sets what the model absorbed statistically, not what it can look up; adding a document to a RAG corpus is not adding it to the model. A knowledge cutoff describes the pretraining corpus, not the system's access to current information.

When to use it

Reach for it when:

• Judging a model's likely coverage of a domain, language or code ecosystem
• Reasoning about cutoffs, staleness, and where retrieval is mandatory
• Planning continued pretraining, where composition and deduplication dominate the outcome

Reach for something else when:

• Treating undisclosed composition as if it were known — most frontier recipes are not published
• Assuming presence in pretraining means reliable recall of a specific fact
• Comparing token counts across models as a quality ranking

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pretraining Data — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path