ModelRefs / Tokens Per Second (TPS) — AI Glossary
Tokens Per Second (TPS) — AI Glossary
The decode throughput of an inference server; measures how fast generated tokens are produced after the first token. Also called TPS or decode throughput.
Overview
TPS (also inter-token latency: 1/TPS seconds between tokens) is the key streaming smoothness metric. Determined by model size, hardware memory bandwidth, and batch size. H100 serving LLaMA-3-70B typically achieves 30–80 TPS per request. Human reading speed ~4–6 tokens/s, so >20 TPS is imperceptible for chat.
Reference details
| Topic | inference |
|---|---|
| Also known as | TPS, decode throughput, generation speed |
| Last reviewed | 2026-06-24 |
Related terms
Example: Two numbers, and only one of them users feel
At 40 tokens per second the gap between tokens is 1 / 40 = 25 milliseconds, and a 600-token answer takes 600 / 40 = 15 seconds to finish. Comfortable reading runs around 5 tokens per second, so anything above roughly 20 TPS arrives faster than a person can read it — pushing 40 to 80 buys nothing a user can perceive on a streamed chat response. Where it does matter is when nobody is reading: an agent loop consuming its own output, or a batch job, is throughput-bound, and there the number is the cost.
Commonly confused with
Tokens per second is decode throughput after generation starts; time to first token is the wait before it. They are governed by different things — TTFT by prompt length and queueing, TPS by memory bandwidth and batch size — and a system can be excellent at one and poor at the other. Per-request TPS also differs from server aggregate TPS, which rises with batching while each individual request slows.
When to use it
Reach for it when:
- Sizing hardware for a target concurrency, where aggregate throughput is the constraint
- Agentic and batch workloads, where output feeds a machine rather than a reader
- Comparing serving engines on identical hardware, model and batch settings
Reach for something else when:
- Optimising perceived speed of chat — past roughly 20 TPS, TTFT is the lever
- Comparing published figures without the hardware, quantization and batch size
- As a quality proxy: a faster model is not a better one
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tokens Per Second (TPS) — AI Glossary.
Frequently asked questions
What is Tokens Per Second (TPS)?
The decode throughput of an inference server; measures how fast generated tokens are produced after the first token.
Is Tokens Per Second (TPS) the same as TPS?
Yes — TPS, decode throughput, generation speed are common aliases for Tokens Per Second (TPS).
What concepts are related to Tokens Per Second (TPS)?
Closely related concepts include time to first token, throughput, gpu memory.