ModelRefs / Continuous Batching — AI Glossary
Continuous Batching — AI Glossary
An inference scheduling technique that swaps finished sequences out of the batch and new requests in at every decoding step.
Overview
Continuous batching keeps GPUs fully utilized under bursty request patterns — the key reason vLLM and SGLang achieve high throughput. Contrasts with static batching, which waits for all requests in a batch to finish before accepting new ones.
Reference details
| Topic | inference |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Commonly confused with
Static batching waits for every sequence in a batch to finish, so one long generation holds short ones hostage and the slot sits idle. Continuous batching evicts a sequence the moment it completes and admits a waiting request in its place, at every decoding step. The gain is largest exactly where static batching is worst: mixed workloads with widely varying output lengths.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Continuous Batching — AI Glossary.
Frequently asked questions
What is Continuous Batching?
An inference scheduling technique that swaps finished sequences out of the batch and new requests in at every decoding step.
What concepts are related to Continuous Batching?
Closely related concepts include throughput, vllm, paged attention.