RAG (Retrieval-Augmented Generation)
A technique that gives an LLM access to external knowledge by retrieving relevant documents before generating. RAG combines search (find relevant info) with generation (LLM writes the answer). Reduces hallucination and lets the model use private or recent data. The standard pattern for production LLM apps.
- Q&A over your documents.
- Chatbots that cite sources.
- Any LLM app needing accurate, sourced, or private data.
- Tasks where the LLM already knows enough.
- Real-time data that changes faster than you can index.
Definition
RAG retrieves relevant documents from a knowledge base (vector database), adds them to the LLM prompt as context, and the LLM generates an answer grounded in those documents. Steps: embed the query, search a vector DB for similar chunks, prepend retrieved chunks to the prompt, generate.
Why it exists
LLMs hallucinate and have a knowledge cutoff. RAG grounds answers in retrieved facts, reducing hallucination and enabling use of private/recent data without retraining the model.
How it works
1. Chunk your documents. 2. Embed each chunk (text → vector). 3. Store vectors in a vector DB (Pinecone, pgvector). 4. At query time, embed the query, search for similar chunks (cosine similarity). 5. Prepend retrieved chunks to the LLM prompt. 6. LLM generates an answer citing the context.
Scaling limits
Retrieval quality limits answer quality (garbage in, out). Vector DB cost at scale. Chunking strategy matters. Context window limits how many chunks fit. Latency of embedding + search + generation.
Cost shape
Vector DB (free pgvector to paid Pinecone). Embedding API costs. LLM inference costs. Storage for embeddings.
Alternatives
- Fine-tuning — Bakes knowledge into the model, but expensive and hard to update.
- Long context (stuff all docs in prompt) — Simple but expensive and hits context limits.
Failure modes
- Poor retrieval (wrong chunks).
- Chunking too big/small.
- Stale index.
- Context window overflow.
- LLM ignoring retrieved context.