Skip to content

Workflow · Productivity

Build an AI Docs Chatbot

A RAG-powered chatbot that answers questions from your documentation with citations — the kind every SaaS site needs in 2026.

Intermediate3–5 hoursStack: Claude or GPT-5, OpenAI embeddings, pgvector or Qdrant, Next.js
RAGChatbotDocsProduction
Edited by The AIKnowHub team · Editorial team

Problem

Users can't find what they need in your docs. Search is keyword-only. Support handles the same questions over and over. Meanwhile, the answer exists — somewhere in your knowledge base.

Final output

An embedded chat widget on your docs site. User asks a question; the bot answers in 2–4 sentences with inline citations linking to the relevant doc pages. Logs every conversation for analysis.

Architecture

Docs (Markdown / HTML / Notion)
  → Chunk + embed (one-time + on update)
  → pgvector / Qdrant index
  → User question
    → Embed query → Retrieve top-5 chunks
    → Send to LLM with system prompt + chunks + question
    → Stream answer with [n] citations
    → Log for analysis

Step-by-step

  1. 01

    Extract and chunk docs

    Parse Markdown / HTML / Notion. Chunk on semantic boundaries (headings, paragraphs). 400–800 tokens per chunk, 10% overlap.

  2. 02

    Embed and index

    Use text-embedding-3-small or voyage-3-lite. Store vectors + metadata (URL, heading, last-updated) in pgvector.

  3. 03

    Build the retrieval endpoint

    Embed the user's query, run vector search (top-10), optionally rerank to top-5. Return chunks + URLs.

  4. 04

    Generate cited answers

    Send chunks + question to the LLM with a strict prompt: cite every claim, refuse if no relevant chunks. Stream tokens to the client.

  5. 05

    Add the chat UI

    Embed a floating chat widget. Show citations as inline numbered links. Log every Q&A pair to your DB.

Why this matters

Every modern SaaS docs site needs an AI assistant. Users expect it. Support teams need it. And the technology has gotten reliable enough that a weekend of work produces something genuinely useful.

This is the most-built workflow in production AI in 2026. Worth knowing how to do well.

The chunking decision

Most docs bot quality is determined at chunking time. Heuristics:

  • Markdown — chunk on H2 boundaries, with H1 + path as metadata.
  • API references — one endpoint per chunk.
  • Long-form articles — 600–800 tokens per chunk with 10% overlap.
  • Code-heavy docs — keep code blocks intact; never split mid-block.

Run an eval after chunking: ask 20 real questions, check if the right chunks come back as top-5. If they don't, fix chunking before touching anything else.

Hybrid search > vector-only

Vector search misses exact-term matches (product names, error codes, version numbers). Layer in BM25:

Top 30 from BM25 + Top 30 from vectors
  → Dedupe
  → Rerank top 5 with cross-encoder
  → Send to LLM

10–20% retrieval-quality improvement for moderate effort.

The "I don't know" prompt

Most docs bots fail by hallucinating. Fix it explicitly:

If the documentation does not contain the information needed to answer this question, do NOT guess. Say: "I don't see this in our docs. You can ask the community at {{url}}."

This single instruction kills a huge category of failures.

Logging is the second product

Every question users ask is signal:

  • Most-asked = best candidates for FAQ pages.
  • "I don't know" answers = docs gaps.
  • Failed retrievals = chunking issues.
  • Highly-rated answers = good content patterns to replicate.

A weekly review of the top 50 questions is the single highest-ROI activity for the team that owns docs.

Prompt examples

Copy any of these, replace the placeholders, run.

System prompt for docs bot

You are the official AI assistant for {{product}} documentation.

Rules:
- Answer ONLY using the documentation chunks provided below. Do not use general knowledge.
- Every factual claim must cite a chunk with [n] where n is the chunk number.
- If the docs don't contain the answer, say: "I don't see that covered in our docs. You can ask in our community at {{url}}."
- Keep answers concise (2–4 sentences) unless the question explicitly asks for detail.
- If the question is ambiguous, ask one clarifying question instead of guessing.

Documentation chunks:
{{chunks}}

Question: {{question}}

Reranker prompt (optional)

Given the user question and 10 candidate chunks, rank the chunks by relevance to answering the question.

Output: { "ranked_ids": [chunk_id, chunk_id, ...] } with the top 5 most relevant first.

Question: {{question}}
Chunks: {{candidates}}

Cost estimate

Line itemApprox. cost
Initial embedding (50K chunks)$1.00
Per query embedding$0.0001
Per query LLM (Sonnet)$0.02
Per query reranker (optional)$0.005
Per 1K queries (typical)$25.00
Total per run (approx.)~$26.03

Costs depend on model choice, content length, and how aggressively you cache.

Optimizations

  • Cache the system prompt with prompt caching — 75–90% input cost reduction.
  • Batch embedding updates — only re-embed changed pages.
  • Add hybrid search (BM25 + vectors) for ~15% recall improvement on technical terms.
  • Use a cheap model (Haiku / Mini) for the answer generation if quality holds — cuts cost 5x.
  • Pre-compute embeddings for the most common queries; serve cached answers when match score is very high.

Common mistakes

  • Chunking on character count instead of semantic boundaries — destroys context.
  • Skipping citations — users don't trust uncited LLM answers; engagement collapses.
  • Forgetting to handle 'I don't know' — model hallucinates rather than refusing.
  • Indexing once, never updating — docs drift, retrieval quality silently rots.
  • No conversation logging — no way to find what's failing or where the docs are weakest.

Frequently asked questions

Webhook from your docs CMS / repo on commit → trigger re-embedding for changed pages. Don't re-index everything daily — that's wasteful.

Related on AIKnowHub