ModelRefs / NVIDIA H100 — AI Glossary
NVIDIA H100 — AI Glossary
NVIDIA's 2022 data-center GPU with 80 GB HBM3, 3.35 TB/s memory bandwidth, and FP8 Transformer Engine; the dominant LLM training and inference chip.
Overview
The H100 SXM5 (80 GB) offers 989 TFLOPS BF16 and 3.35 TB/s HBM3 bandwidth. Its Transformer Engine accelerates attention and matrix ops with FP8 precision. Standard cluster size is 8× H100 NVLink nodes. Successor H200 upgrades to HBM3e (141 GB). Reserved on AWS (p4de/p5), Azure (ND H100 v5), and GCP (A3).
Reference details
| Topic | inference |
|---|---|
| Also known as | H100 GPU, H100 SXM |
| Last reviewed | 2026-06-24 |
Related terms
Example: Decoding is bandwidth-bound, and the ceiling is arithmetic
Generating each token requires reading every weight the model uses. A 70-billion-parameter model at fp16 is about 140 GB, which already needs two 80 GB cards. At roughly 3.35 TB/s of memory bandwidth, moving 140 GB takes 140 / 3350 ≈ 0.042 seconds — about 42 ms per token, or a ceiling near 24 tokens per second for a single unbatched request, before any compute is considered. Nothing about the card's arithmetic throughput changes that number. This is why quantization helps decode speed so directly: halving the bytes read halves the floor.
Commonly confused with
Peak FLOPS describes prefill, which processes the whole prompt in parallel and is compute-bound. Decode is memory-bound, one token at a time. A card can be excellent at one and unremarkable at the other, which is why comparing accelerators on headline FLOPS predicts training throughput far better than it predicts interactive generation speed.
When to use it
Reach for it when:
- Capacity planning, where bandwidth and memory set the achievable tokens per second
- Deciding how many cards a model needs before quantization is considered
- Understanding why batching raises aggregate throughput without speeding any single request
Reach for something else when:
- Estimating decode speed from FLOPS figures
- Assuming a newer card is proportionally faster at generation — check bandwidth, not compute
- Sizing from weights alone: the KV cache grows with context and concurrency and must be budgeted
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to NVIDIA H100 — AI Glossary.
Frequently asked questions
What is NVIDIA H100?
NVIDIA's 2022 data-center GPU with 80 GB HBM3, 3.35 TB/s memory bandwidth, and FP8 Transformer Engine; the dominant LLM training and inference chip.
Is NVIDIA H100 the same as H100 GPU?
Yes — H100 GPU, H100 SXM are common aliases for NVIDIA H100.
What concepts are related to NVIDIA H100?
Closely related concepts include gpu memory, tpu, gpu utilization.