ModelRefs / What is RAG? — Tutorial
What is RAG? — Tutorial
Retrieval-Augmented Generation — give any LLM access to a knowledge base without retraining. Covers Why RAG exists, The RAG pipeline.
Overview
Retrieval-Augmented Generation — give any LLM access to a knowledge base without retraining
Level: Intermediate. Estimated reading time: 20 minutes.
Why RAG exists
LLMs have two fundamental limitations: their knowledge has a training cutoff date, and they cannot access proprietary internal documents. RAG (Lewis et al., 2020) solves both.
The core idea: before generating an answer, retrieve relevant documents from an external knowledge base and include them in the prompt. The LLM reads the retrieved context and answers from it — grounding the response in real, up-to-date sources.
Why RAG beats fine-tuning for knowledge: fine-tuning bakes knowledge into weights, making it hard to update or remove. RAG keeps knowledge in a database you can update in real time. You can also cite sources, which matters for trust.
RAG vs long context: you could just put your entire knowledge base in the context window. For small knowledge bases (<100k tokens) this works. For millions of documents, retrieval is necessary. RAG also lets you use smaller, cheaper models since you don't need massive context windows.
The RAG pipeline
A RAG system has two phases:
Indexing (offline): split documents into chunks (typically 200–500 tokens with overlap), embed each chunk using an embedding model (e.g. text-embedding-3-small), store (chunk text + embedding) in a vector database.
Querying (online): 1. Embed the user's query with the same embedding model 2. Retrieve the top-k most similar chunks by cosine similarity 3. Inject retrieved chunks into the prompt: "Answer using only the context below: [chunks] | Question: [query]" 4. Generate the final answer with the LLM
The retrieved chunks are the "grounding" that prevents hallucination. If the answer isn't in the retrieved context, a well-prompted RAG system should say so.
Key tuning knobs: chunk size (smaller = more precise, larger = more context per chunk), top-k (more chunks = more recall, longer prompt), overlap (prevents answers split across chunk boundaries), reranking (use a cross-encoder to re-score top-20 candidates and keep top-5).
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to What is RAG? — Tutorial.