ModelRefs / Embedding Dimension — AI Glossary
Embedding Dimension — AI Glossary
The size of the vector space in which tokens are represented inside a transformer; a key hyperparameter controlling model width.
Overview
Also called d_model or hidden size. Typical values: GPT-2 small=768, LLaMA-7B=4096, GPT-4 ~12,288 (estimated). Larger embedding dimensions increase parameter count and representational capacity quadratically. It interacts with the number of attention heads (each head uses d_model/n_heads dimensions).
Reference details
| Topic | architecture |
|---|---|
| Also known as | hidden size, d_model, model dimension |
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Embedding Dimension — AI Glossary.
Frequently asked questions
What is Embedding Dimension?
The size of the vector space in which tokens are represented inside a transformer; a key hyperparameter controlling model width.
Is Embedding Dimension the same as hidden size?
Yes — hidden size, d_model, model dimension are common aliases for Embedding Dimension.
What concepts are related to Embedding Dimension?
Closely related concepts include parameter count, multi head attention, vocabulary size.