ModelRefs / Build a Production RAG Stack — Tutorial

Build a Production RAG Stack — Tutorial

Assemble a Retrieval-Augmented Generation pipeline: chunking, embeddings, vector store, retrieval evaluation, and grounded generation. Covers What is RAG.

Overview

Assemble a Retrieval-Augmented Generation pipeline: chunking, embeddings, vector store, retrieval evaluation, and grounded generation.

Level: Expert. Estimated reading time: 75 minutes.

What is RAG?

Retrieval-Augmented Generation (RAG) grounds an LLM's answers in real documents. Instead of relying on what the model memorized during training, RAG retrieves relevant passages from your own knowledge base at query time — and feeds them to the model as context.

This solves two core LLM problems: hallucination (the model inventing facts) and knowledge cutoffs (training data going stale).

The RAG Architecture

A production RAG stack has three stages:

1. INDEXING (offline, runs once or on update) • Load documents (PDF, HTML, Markdown, database rows) • Chunk — split into overlapping text windows (typically 512–1024 tokens) • Embed — convert each chunk to a dense vector using an embedding model • Store — upsert vectors + metadata into a vector database

2. RETRIEVAL (online, per query) • Embed the user's query with the same model • ANN search — find top-k chunks by cosine / dot-product similarity • Re-rank — optionally pass top-k through a cross-encoder for precision

3. GENERATION (online, per query) • Build prompt: system instructions + retrieved chunks + user query • Call LLM — generates a grounded answer citing retrieved sources • Stream response back to user

Chunking Strategy

Chunking is the most impactful decision in RAG — bad chunks kill retrieval quality.

Rules of thumb: • Chunk size 512–1024 tokens balances context richness vs. retrieval precision • 10–20% overlap between chunks prevents context from being cut mid-sentence • Respect document boundaries: chunk within a section, not across chapters • For code: chunk by function/class, not line count • For tables: keep table rows together, include column headers in each chunk

Embedding Model Selection

The embedding model converts text into vectors that retrieval searches over. The choice constrains retrieval quality, so treat it as a decision to evaluate rather than a default to inherit.

Compare candidates on the dimensions that actually change your result: • Language coverage — does it handle the languages in your corpus, not just English? • Retrieval modes — dense only, or also sparse and multi-vector? More modes can help recall but add index size, latency, and operational complexity. • Sequence length — can it embed your chunk sizes without truncation? • Deployment path — hosted API (pay per token, provider terms) or open weights you self-host (you own compute, scaling, and monitoring). • Cost and latency at your real query volume, not at demo scale.

Two candidates with published ModelRefs references: text-embedding-3-large is a hosted OpenAI embedding model, and BGE-M3 is an open-weight multilingual retrieval model from BAAI that exposes dense, sparse, and multi-vector representations in a single artifact. BGE-M3 is a retrieval and embedding component, not a generator, and its ModelRefs profile is Provisional with limited benchmark coverage — provider-reported multilingual results do not establish quality on your corpus. Check each model's page for its current specifications, evidence status, and limitations rather than relying on figures repeated here.

Whichever you shortlist, evaluate on your actual domain and languages. General leaderboard rankings do not reliably transfer to a specific corpus, chunking strategy, or query mix — see the retrieval recall and grounding tutorial for how to measure that.

Retrieval Evaluation

You can't trust a RAG system you haven't measured. Key metrics:

• Recall@k — what fraction of queries have the correct chunk in top-k results? • MRR (Mean Reciprocal Rank) — how high does the correct chunk rank? • Faithfulness — does the generated answer match the retrieved context? (LLM-as-judge) • Answer Relevance — is the answer actually answering the question?

Tools: RAGAS (Python), LangChain Evaluators, Promptfoo

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Build a Production RAG Stack — Tutorial.