ModelRefs / FSDP (Fully Sharded Data Parallel) — AI Glossary

FSDP (Fully Sharded Data Parallel) — AI Glossary

PyTorch's native distributed training strategy sharding model parameters, gradients, and optimizer states across all workers.

Overview

FSDP (PyTorch 1.12+) implements ZeRO-3-style sharding natively in PyTorch, replacing Fairscale. Each worker holds only 1/N of parameters; full parameters are all-gathered only for the forward/backward pass then re-sharded. Now the standard path for large-scale PyTorch training, used for LLaMA training and fine-tuning.

Reference details

Topic	training
Last reviewed	2026-06-24

Example: The memory you save is bought with network traffic

Sharding puts 1/N of the parameters on each of N workers, which is what makes a model too large for one device trainable. But the maths still needs whole layers: before each layer's forward pass the workers all-gather its full parameters, use them, then discard the copy and re-shard. Every layer, every step, in both directions. That is why sharded training is sensitive to interconnect in a way that plain data parallelism is not — the same job that runs well on fast intra-node links can become communication-bound across slower ones, with the GPUs idle waiting.

Commonly confused with

FSDP is PyTorch's native implementation of the same idea as DeepSpeed ZeRO stage 3, not a competing technique. Choosing between them is mostly an ecosystem and tooling decision. Neither is a substitute for parameter-efficient fine-tuning: if the optimizer state is the problem and full-model quality is not required, LoRA removes the problem instead of distributing it.

When to use it

Reach for it when:

Full fine-tuning or pretraining where optimizer state and gradients exceed device memory
PyTorch-native stacks, where it avoids an extra dependency and config surface
Multi-node runs on fast interconnect, where the all-gather traffic is affordable

Reach for something else when:

Anything LoRA or another PEFT method can do — far less machinery for the same outcome
Slow interconnects, where communication dominates and utilisation collapses
Inference, which has different sharding needs and dedicated engines

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to FSDP (Fully Sharded Data Parallel) — AI Glossary.

Frequently asked questions

What is FSDP (Fully Sharded Data Parallel)?