Model Parallelism — AI Glossary

Reference details

Related terms

• Distributed Inference
• Pipeline Parallelism
• DeepSpeed

Example: Two ways to split, with different network bills

Tensor parallelism splits individual weight matrices across devices, so every device holds a slice of every layer and they must synchronise within each layer — with roughly two collective operations per transformer block, a deep model reaches hundreds of synchronisation points per forward pass. That demands very fast links, which is why it is normally kept inside one node. Pipeline parallelism instead assigns whole layers to different devices and passes activations forward point-to-point: far less traffic, tolerant of slower links, but it introduces idle time while later stages wait for earlier ones. Large runs combine both, plus data parallelism, precisely because they fail in different directions.

Commonly confused with

Model parallelism splits the model because it does not fit; data parallelism replicates the model and splits the batch. Sharded strategies like FSDP and ZeRO look like model parallelism but are data-parallel underneath — they shard storage and reconstruct full layers on demand, rather than permanently distributing the computation.

When to use it

Reach for it when:

• Models whose weights exceed a single device, where sharding is the only option
• Tensor parallelism within a node on fast interconnect; pipeline parallelism across nodes
• Serving very large models, where the alternative is not serving them

Reach for something else when:

• Models that fit on one device — the communication is pure overhead
• Tensor parallelism across slow links, where synchronisation dominates
• As a first response to memory pressure: quantization and KV-cache limits are cheaper

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Parallelism — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path