ModelRefs / Megatron-LM — AI Glossary
Megatron-LM — AI Glossary
NVIDIA's framework for training large transformer models with 3D parallelism (data, tensor, pipeline), used for frontier model training.
Overview
Megatron-LM (Shoeybi et al. 2019) pioneered tensor parallelism for transformers, enabling training of 530B-parameter models. Its 3D parallelism combines data-parallel (FSDP), tensor-parallel (intra-layer), and pipeline-parallel (inter-layer) strategies. Used by NVIDIA, Microsoft (Turing-NLG), and as the upstream for many LLaMA training implementations.
Reference details
| Topic | infrastructure |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Primary source
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Megatron-LM — AI Glossary.
Frequently asked questions
What is Megatron-LM?
NVIDIA's framework for training large transformer models with 3D parallelism (data, tensor, pipeline), used for frontier model training.
What concepts are related to Megatron-LM?
Closely related concepts include deepspeed, fsdp, model parallelism.