ModelRefs / Vector Databases — Tutorial
Vector Databases — Tutorial
How Pinecone, Chroma, and Weaviate store billions of embeddings and retrieve the nearest in milliseconds. Covers Why you need a vector database.
Overview
How Pinecone, Chroma, and Weaviate store billions of embeddings and retrieve the nearest in milliseconds
Level: Advanced. Estimated reading time: 35 minutes.
Why you need a vector database
A numpy array of 1M embeddings at 1536 dimensions takes 6GB of RAM and requires O(n) brute-force search per query. At 100M vectors, brute-force is too slow for real-time queries.
Vector databases solve this with Approximate Nearest Neighbour (ANN) indexes. ANN trades a small accuracy loss for orders-of-magnitude faster search. Common ANN algorithms:
HNSW (Hierarchical Navigable Small World): builds a multi-layer graph where each node connects to its nearest neighbours. Search starts at the top layer (few nodes, long-range connections) and navigates down. Query time is O(log n). This is the most popular production algorithm.
IVF (Inverted File Index): cluster vectors into k buckets using k-means. Search only probes the nearest few buckets (n_probe parameter). Much faster than brute-force but misses vectors in non-probed buckets.
Product Quantisation (PQ): compress vectors by splitting them into subvectors and encoding each with a codebook. Reduces memory 8–64× at the cost of precision.
Popular vector databases compared
Chroma: open-source, in-process Python library. Perfect for prototyping — no server needed. Scales to millions of vectors but not billions. pip install chromadb.
Pinecone: managed cloud service. Handles scaling, replication, and updates. Serverless tier starts free. SDK: pip install pinecone-client.
Weaviate: open-source, self-hostable or cloud. Combines vector search with filtered keyword search (hybrid). Strong at multi-modal (text + image embeddings together).
pgvector: PostgreSQL extension. Add vector search to an existing Postgres database. Ideal when you already have relational data and don't want another system. Uses IVF-Flat or HNSW indexes. pip install psycopg2.
Qdrant: open-source, Rust-based, very fast. Supports payload filtering alongside vector search.
For production RAG: Pinecone or Weaviate. For prototyping: Chroma. For existing Postgres users: pgvector.
Metadata filtering and hybrid search
Pure semantic search isn't always enough. "Find documents about AI safety written after 2023" requires both semantic similarity and a date filter.
Metadata filtering: store metadata alongside each vector (author, date, source, category). At query time, filter by metadata first, then do ANN search within the filtered subset. All major vector DBs support this.
Hybrid search: combine dense vector search with sparse keyword search (BM25). A query like "Apple M3 benchmark" needs keyword matching ("M3", "Apple") AND semantic understanding. Most production RAG systems use hybrid search.
Reranking: retrieve top-50 by ANN (fast, approximate), then re-rank with a cross-encoder (slow, precise) to get the true top-5. The cross-encoder sees both the query and each document together, capturing fine-grained relevance that embedding cosine similarity misses.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Vector Databases — Tutorial.