Chunking Strategies for RAG

Why chunking matters more than your embedding model

RAG has two phases: index time and query time. At index time, you split documents into chunks, embed each chunk, and store the vectors. At query time, you retrieve the top-k chunks and paste them into the prompt.

The retriever can only return what you chunked. If the answer spans two chunks and you split in the middle, neither chunk will rank highly. If you chunk a pricing table row-by-row, the retriever finds individual prices but loses the column headers. Chunking is the ceiling on retrieval quality — everything downstream (embeddings, vector DB, reranking, LLM) operates within that ceiling.

Most teams spend weeks tuning embedding models while their chunks are broken. Don't be that team.

The core tradeoff

Every chunking decision balances two forces:

There's no universal optimum. The right size depends on your documents and your users' questions.

Strategy 1: Fixed-size with overlap

The simplest approach: split every N tokens, with M tokens of overlap between consecutive chunks.

Chunk 1: tokens 0–512
Chunk 2: tokens 448–960    ← 64-token overlap
Chunk 3: tokens 896–1408

Defaults: 512 tokens, 64-token overlap (~12%).

Pros: Fast, predictable, works for homogeneous prose. Cons: Splits mid-sentence, mid-paragraph, mid-table. Ignores document structure.

When to use: Blog posts, articles, general documentation. Your first implementation.

Implementation tip: Split on sentence boundaries when possible. Libraries like LangChain's RecursiveCharacterTextSplitter try \n\n → \n → . → before falling back to character count.

Strategy 2: Structure-aware chunking

Respect the document's own hierarchy — headers, sections, list items, code blocks.

Markdown / HTML docs:

• Split on ## or ### headers.
• Each section becomes one chunk (or sub-split if a section exceeds your max).
• Store the header trail as metadata: ["Getting Started", "Authentication", "API Keys"].

Code:

• Split on function, class, or module boundaries.
• Include the file path, function signature, and imports in metadata.
• Keep related tests in the same chunk when possible.

Q&A / FAQ docs:

• Each question-answer pair is exactly one chunk.
• Don't merge multiple Q&As — users ask one question at a time.

Tables:

• Never split a table across chunks.
• If the table is too large, include column headers in every sub-chunk or store the full table as a single chunk regardless of size.

Strategy 3: Semantic chunking

Use an embedding model to detect topic boundaries. Embed sentences sequentially; when the similarity between consecutive sentences drops below a threshold, start a new chunk.

Pros: Chunks align with natural topic shifts. Great for long unstructured prose (research papers, transcripts). Cons: Slow at index time (one embedding per sentence). Produces uneven chunk sizes. Threshold tuning is finicky.

When to use: Podcast transcripts, research papers, long-form reports where structure-aware splitting fails.

Strategy 4: Parent-child chunking

The most impactful upgrade for production RAG:

1. Child chunks (128–256 tokens) — embedded and searched. Tight match to queries.
2. Parent chunks (full section, 1000–2000 tokens) — not embedded. Retrieved by association when a child matches.

Query: "How do I rotate API keys?"
  → Child chunk matched: "...click Settings > API Keys > Rotate..."
  → Parent chunk passed to LLM: full "API Keys" section with setup, rotation, and revocation

You get precise retrieval and rich context. This is what most mature RAG systems converge on.

Metadata is half the chunking strategy

Every chunk should carry:

• Source — file path, URL, or document ID
• Position — page number, section header, line range
• Type — prose, code, table, faq
• Timestamp — when the source was last updated

Metadata enables pre-filtering at query time: "only search API docs from the last 6 months" beats hoping the embedding model figures it out.

How to evaluate chunking

Don't guess. Measure.

1. Build a retrieval eval set — 20–50 questions with known gold-source passages. See Evals Explained for the framework.
2. Measure recall@k — for each question, is the gold passage in the top-k retrieved chunks?
3. Target 80%+ recall@5 before touching the LLM prompt or switching embedding models.
4. Inspect failures — when recall is low, read the missed chunks. You'll find split boundaries, missing context, or metadata gaps.

Run this eval every time you change chunk size, overlap, or strategy. Chunking regressions are silent and expensive.

Chunking by content type — quick reference

Common failure modes

• Splitting tables row-by-row — retriever finds a cell value but the LLM has no column context.
• No overlap on procedural docs — step 4 is in chunk 2, step 5 is in chunk 3, and neither chunk makes sense alone.
• Chunking code without imports — the function body is retrieved but the LLM doesn't know what types or modules are in scope.
• Stale chunks after doc updates — you re-embed changed docs but forget to delete old chunks, so the retriever serves outdated passages.
• Ignoring document updates — chunking once at deploy and never re-indexing. Pair chunking with a change-detection pipeline.

The decision tree

Is your content structured (headers, sections, endpoints)?
  → Yes: structure-aware chunking
  → No: fixed-size with overlap

Is recall@5 below 80%?
  → Yes: try parent-child chunking
  → No: you're probably fine

Do you have tables or code?
  → Yes: specialized rules (whole table, function boundaries)
  → No: defaults work

Still failing?
  → Build a retrieval eval and inspect the misses

Chunking isn't glamorous. It's the unsexy infrastructure decision that determines whether your docs chatbot actually works. Invest here first.