ModelRefs / Pretraining — AI Glossary
Pretraining — AI Glossary
The initial large-scale training phase where a model learns general language patterns from web-scale data. Fine-tuning adapts a model that already exists.
Overview
Pretraining on trillions of tokens of text (and code, math, images) is what makes foundation models capable. It is extraordinarily compute-intensive — frontier model pretraining runs cost tens to hundreds of millions of dollars.
Reference details
| Topic | training |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Commonly confused with
Pretraining builds general capability from scratch on web-scale data and costs a training run few organisations can fund. Fine-tuning adapts a model that already exists. Continued pretraining sits between them — more next-token training on domain data, without task-specific pairs — and is the right label for the work often mislabelled as fine-tuning when the aim is absorbing a domain rather than learning a task format.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Pretraining — AI Glossary.
Frequently asked questions
What is Pretraining?
The initial large-scale training phase where a model learns general language patterns from web-scale data.
What concepts are related to Pretraining?
Closely related concepts include foundation model, fine tuning, sft.