ModelRefs / GPU Memory (VRAM) — AI Glossary

GPU Memory (VRAM) — AI Glossary

The on-chip memory of a GPU (VRAM) that must hold model weights, KV cache, and activations for LLM inference. VRAM is the binding constraint for LLM inference.

Overview

VRAM is the binding constraint for LLM inference. An A100 80GB holds a 40B float16 model. KV cache grows with batch size × context length × layers × head dimension, often consuming more memory than weights at long context. Strategies: quantization (reduce weight size), paged attention (PagedKV), continuous batching.

Reference details

Topicinference
Also known asVRAM, GPU RAM
Last reviewed2026-06-24

Related terms

Example: Weights are only the first line of the budget

A 13B model at FP16 is about 26 GB of weights. On an 80 GB card that leaves roughly 54 GB — but the KV cache scales with batch size and context, and at long context with real concurrency it can exceed the weights. Capacity planning that stops at weights will over-promise concurrency every time.

weights: 13e9 params x 2 bytes = 26 GB
card   : 80 GB
headroom for KV cache + activations = 54 GB   # shrinks fast with context x batch

Commonly confused with

VRAM is not the same as system RAM and generally cannot borrow from it. Once weights, cache and activations exceed the card you do not degrade gracefully — you fail to allocate, or you offload and lose most of the speed advantage that motivated the GPU.

When to use it

Reach for it when:

  • Choosing hardware for a model and a target concurrency
  • Diagnosing why concurrency stalls well below expectation
  • Deciding whether quantization is necessary rather than optional

Reach for something else when:

  • Sizing from weights alone; the cache is the variable that bites
  • Assuming a bigger card scales concurrency linearly — cache per request bounds it

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GPU Memory (VRAM) — AI Glossary.

Frequently asked questions

What is GPU Memory (VRAM)?

The on-chip memory of a GPU (VRAM) that must hold model weights, KV cache, and activations for LLM inference.

Is GPU Memory (VRAM) the same as VRAM?

Yes — VRAM, GPU RAM are common aliases for GPU Memory (VRAM).

What concepts are related to GPU Memory (VRAM)?

Closely related concepts include kv cache, quantization, h100.