ModelRefs / Mixed Precision Training — AI Glossary
Mixed Precision Training — AI Glossary
Training using low-precision (FP16/BF16) for forward/backward passes and full-precision (FP32) for optimizer states, balancing speed and stability.
Overview
Mixed precision (NVIDIA AMP, PyTorch autocast) halves memory and doubles throughput versus pure FP32 by storing activations and weights in 16-bit. A FP32 master copy of weights is maintained for stable optimizer updates. Loss scaling (for FP16) or BF16 (no scaling needed) prevents gradient underflow. Universal practice for modern LLM training.
Reference details
| Topic | training |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Mixed Precision Training — AI Glossary.
Frequently asked questions
What is Mixed Precision Training?
Training using low-precision (FP16/BF16) for forward/backward passes and full-precision (FP32) for optimizer states, balancing speed and stability.
What concepts are related to Mixed Precision Training?
Closely related concepts include bf16, gradient checkpointing, deepspeed.