ModelRefs / FlashAttention — AI Glossary
FlashAttention — AI Glossary
An IO-aware exact-attention algorithm that tiles attention computation to fit in GPU SRAM, eliminating the quadratic memory traffic of standard attention.
Overview
FlashAttention-2 and FlashAttention-3 are the default attention kernels in vLLM, TGI, and PyTorch SDPA. Achieves 2–10× speedup on long contexts at zero accuracy cost by minimizing reads/writes to slow HBM memory.
Reference details
| Topic | inference |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Commonly confused with
The counterpart to PagedAttention: this one changes the computation, tiling it to keep intermediate results in fast on-chip memory instead of writing the full attention matrix out. It is exact, not an approximation — the outputs are the same as standard attention, which distinguishes it from sparse or windowed variants that deliberately drop pairs to save work.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to FlashAttention — AI Glossary.
Frequently asked questions
What is FlashAttention?
An IO-aware exact-attention algorithm that tiles attention computation to fit in GPU SRAM, eliminating the quadratic memory traffic of standard attention.
What concepts are related to FlashAttention?
Closely related concepts include attention, kv cache, transformer.