ModelRefs / Streaming (Token Streaming) — AI Glossary
Streaming (Token Streaming) — AI Glossary
Delivering model output token-by-token as it is generated, rather than waiting for the full response before sending. Streaming changes delivery, not generation.
Overview
Streaming dramatically improves perceived latency — users see output start immediately. Implemented via server-sent events (SSE) or WebSockets. All major providers (OpenAI, Anthropic, Google) support streaming. Streaming is incompatible with token budget-based pricing that counts output only after completion.
Reference details
| Topic | inference |
|---|---|
| Also known as | token streaming, SSE streaming |
| Last reviewed | 2026-06-24 |
Related terms
Example: The wait you remove is most of the wait
A 400-token answer generated at 50 tokens per second takes 8 seconds to finish. Without streaming the user sees nothing for 8 seconds. With streaming they see the first words in a few hundred milliseconds and read while the rest arrives — and since comfortable reading is well under 50 tokens per second, they never catch up to the stream. Total time is identical. Time to first token is the number that changed, and it is the one users experience as speed.
Commonly confused with
Streaming changes delivery, not generation. It does not reduce cost, total latency, or tokens produced. It also does not make a slow model fast — if time to first token is already seconds, streaming reveals the delay rather than hiding it.
When to use it
Reach for it when:
- Any interactive surface where a human reads the output as prose
- Long responses, where the perceived gain grows with output length
- Agent traces you want shown progressively, so users can interrupt a wrong direction
Reach for something else when:
- Output that must be validated as a whole before display — malformed JSON cannot be unsent
- Guardrail or moderation gates on the completion, which need the full text to decide
- Batch and background jobs, where nobody is waiting and streaming only complicates the client
Referenced by
This term is used by the following ModelRefs references:
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Streaming (Token Streaming) — AI Glossary.
Frequently asked questions
What is Streaming (Token Streaming)?
Delivering model output token-by-token as it is generated, rather than waiting for the full response before sending.
Is Streaming (Token Streaming) the same as token streaming?
Yes — token streaming, SSE streaming are common aliases for Streaming (Token Streaming).
What concepts are related to Streaming (Token Streaming)?
Closely related concepts include latency, inference cost.