ModelRefs / Distributed Inference — AI Glossary
Distributed Inference — AI Glossary
Running a single model inference across multiple GPUs to fit large models or improve throughput beyond a single device's capacity.
Overview
For models exceeding single-GPU VRAM, distributed inference splits weights using tensor parallelism, pipeline parallelism, or both. Tools: vLLM (tensor parallel), DeepSpeed-Inference, TensorRT-LLM. Communication overhead (all-reduce, P2P transfers) constrains scaling efficiency, making NVLink or InfiniBand interconnects critical.
Reference details
| Topic | inference |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Distributed Inference — AI Glossary.
Frequently asked questions
What is Distributed Inference?
Running a single model inference across multiple GPUs to fit large models or improve throughput beyond a single device's capacity.
What concepts are related to Distributed Inference?
Closely related concepts include pipeline parallelism, model parallelism, gpu memory.