ModelRefs / Triton Inference Server — AI Glossary
Triton Inference Server — AI Glossary
NVIDIA's production inference server supporting multiple ML frameworks with GPU batching, model ensembles, and gRPC/HTTP APIs.
Overview
Triton (NVIDIA) serves models from PyTorch, TensorFlow, TensorRT, ONNX, and custom backends. Key features: dynamic batching, model ensembles (chain models into pipelines), concurrent model execution, model analyzer for performance tuning. Used by Microsoft Azure AI and major cloud providers for high-throughput LLM serving.
Reference details
| Topic | infrastructure |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Triton Inference Server — AI Glossary.
Frequently asked questions
What is Triton Inference Server?
NVIDIA's production inference server supporting multiple ML frameworks with GPU batching, model ensembles, and gRPC/HTTP APIs.
What concepts are related to Triton Inference Server?
Closely related concepts include ray serve, bentoml, vllm.