ModelRefs / Speculative Decoding — AI Glossary

Speculative Decoding — AI Glossary

An inference technique where a small draft model proposes tokens that a large verifier model accepts or rejects in parallel.

Overview

Speculative decoding achieves 2–4× throughput at zero quality loss when the draft acceptance rate is high. Used by vLLM, TensorRT-LLM, and most modern inference stacks. The draft model is typically 10–20× smaller than the verifier.

Reference details

Topic	inference
Last reviewed	2026-06-24

Example: Count the tokens you get per expensive pass

A small draft model proposes 4 tokens; the large model verifies them in one pass. If each proposed token is accepted with probability 0.8, the expected run before the first rejection is 0.8 + 0.8² + 0.8³ + 0.8⁴ = 0.8 + 0.64 + 0.512 + 0.41 ≈ 2.36, and the verifier contributes one correct token regardless — roughly 3.4 tokens per expensive pass instead of 1. The critical property is that the output distribution is provably unchanged: rejected drafts are resampled from the large model, so this is not an approximation and costs no quality. The risk is the other direction — a poorly matched draft model with a low acceptance rate makes you pay for drafting and get almost nothing back.

Commonly confused with

Speculative decoding is lossless, which separates it from quantization, distillation and early exit — all of which trade quality for speed. It also is not batching: batching raises aggregate throughput while individual requests get no faster, whereas speculation reduces latency for a single request. The two compose, but they solve different problems.

When to use it

Reach for it when:

Latency-sensitive single-request serving, where batching cannot help
Where a small model from the same family and tokenizer is available as the drafter
Predictable text — code and structured output tend to have high acceptance rates

Reach for something else when:

Throughput-bound batch serving, where the GPU is already saturated
Without measuring the acceptance rate on your own traffic — below roughly break-even it is a net loss
Mismatched draft and target vocabularies, which breaks the verification step

Referenced by

This term is used by the following ModelRefs references:

become llm engineer

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Speculative Decoding — AI Glossary.

Frequently asked questions

What is Speculative Decoding?