ModelRefs / DeepSpeed — AI Glossary
DeepSpeed — AI Glossary
Microsoft's distributed training library implementing ZeRO optimizer stages, pipeline parallelism, and mixed precision to train 100B+ parameter models.
Overview
DeepSpeed ZeRO (Zero Redundancy Optimizer) shards optimizer states, gradients, and parameters across data-parallel workers, eliminating redundant memory. ZeRO-3 shards all three, enabling single-GPU-equivalent memory per device. Used to train BLOOM, Falcon, and Megatron-class models. Integrates with PyTorch DDP and Hugging Face Accelerate.
Reference details
| Topic | training |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: The memory arithmetic ZeRO attacks
Mixed-precision training with Adam holds roughly 16 bytes per parameter: 2 for the fp16 weights, 2 for the fp16 gradients, and 12 for the fp32 master copy plus the two optimizer moments. A 7-billion-parameter model therefore needs about 7e9 × 16 = 112 GB for states alone — before activations, and well beyond a single 80 GB accelerator. ZeRO stage 3 shards all three categories across data-parallel workers, so across 8 GPUs each holds roughly 112 / 8 = 14 GB. The model did not shrink; the redundancy did.
Commonly confused with
DeepSpeed is a library; ZeRO is the sharding technique it introduced. They are not interchangeable — PyTorch FSDP implements the same idea natively, so you can shard without DeepSpeed, and DeepSpeed offers pipeline parallelism and inference kernels that are not ZeRO.
When to use it
Reach for it when:
- Training or full fine-tuning where optimizer state, not the weights, is what exceeds memory
- Multi-node runs needing pipeline and tensor parallelism alongside sharding
- Existing Hugging Face training code, which integrates it through a config file
Reach for something else when:
- Parameter-efficient fine-tuning — LoRA sidesteps the optimizer-state problem entirely
- Inference serving, where a dedicated engine is the better fit
- Single-GPU work: sharding across one worker adds complexity and buys nothing
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to DeepSpeed — AI Glossary.
Frequently asked questions
What is DeepSpeed?
Microsoft's distributed training library implementing ZeRO optimizer stages, pipeline parallelism, and mixed precision to train 100B+ parameter models.
What concepts are related to DeepSpeed?
Closely related concepts include fsdp, zero optimizer, model parallelism.