ModelRefs / Semantic Cache — AI Glossary
Semantic Cache — AI Glossary
A cache that stores LLM responses and retrieves them for semantically similar (not just identical) future prompts. Tools: GPTCache, Zep.
Overview
Semantic caches use embeddings to match incoming prompts against cached queries. Tools: GPTCache, Zep. Most effective for high-repetition use cases (customer support, FAQ). Cache hit rates depend on prompt diversity.
Reference details
| Topic | infrastructure |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Commonly confused with
Frequently confused with prompt caching, and the difference is consequential. Prompt caching reuses computation for an identical prefix and cannot change what the model says. A semantic cache returns a previously stored *answer* because a new question is judged similar — so a similarity threshold that is slightly too loose will confidently serve the answer to a different question. It needs its own correctness evaluation; prompt caching does not.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Semantic Cache — AI Glossary.
Frequently asked questions
What is Semantic Cache?
A cache that stores LLM responses and retrieves them for semantically similar (not just identical) future prompts.
What concepts are related to Semantic Cache?
Closely related concepts include embedding, model gateway, inference cost.