RAG Explained

The problem RAG solves

LLMs are frozen at their training cutoff and they don't know about your private documents, your company wiki, or last week's incident report. Retrieval-Augmented Generation (RAG) fixes that without retraining the model.

The basic loop

1. Index time — Chunk your documents, generate embeddings for each chunk, store them in a vector database.
2. Query time — Embed the user's question, find the top-k most similar chunks, paste them into the prompt as context, then ask the LLM to answer using only that context.

That's it. The "augmentation" is just stuffing relevant snippets into the prompt before the question.

Chunking is where most RAG systems fail

Chunk size matters more than people think:

• Too small — chunks lose context (a paragraph cut in half is meaningless).
• Too large — top-k retrieval surfaces too much noise.
• Naive splitting — splitting on character count breaks tables, code, lists.

Practical defaults: 400–800 tokens per chunk, 10–15% overlap, split on semantic boundaries (paragraphs, headers, code blocks).

For specialized content, customize further:

• Code: split on function boundaries, keep the file path in metadata.
• Tables: keep the whole table as a single chunk, even if large.
• Q&A docs: each Q+A is one chunk.

Retrieval quality

Plain cosine similarity on embeddings often fails. Real systems combine:

• Dense retrieval (embeddings) — captures meaning.
• Sparse retrieval (BM25 / keyword) — captures exact terms.
• Reranking — a small model re-scores the top-50 down to top-5.

This hybrid setup beats embeddings-only by a wide margin on real corpora.

The full pipeline

User question
  → embed query
  → vector search (top-50)
  → BM25 search (top-50)
  → merge + dedupe
  → cross-encoder rerank (top-5)
  → format as context
  → LLM with system prompt + context + question
  → response with citations

You can skip steps for simple cases. Add them as quality demands.

When NOT to use RAG

• The data fits in a single context window — just paste it.
• The user needs exact answers from structured data — query the database directly.
• The task is creative, not factual — retrieval will hurt, not help.

Common failure modes

• Stale index — documents updated but vectors weren't re-embedded.
• Mismatched embedding models at index vs query time.
• No citations in the response, so users can't verify.
• Prompt doesn't actually instruct the model to use only retrieved context — it just gets ignored.

RAG isn't magic. It's just "find the right paragraph, then paste it in." Get the retrieval right and the LLM does the rest.