Case Study · Product
Case Study: RAG Docs Chatbot Shipped in 3 Weeks
How a 12-person B2B SaaS team shipped a cited docs chatbot that deflected 34% of support tickets — without hiring an ML team.
Edited by The AIKnowHub team · Editorial teamResults
Support ticket deflection
34%
Median answer latency
2.1s
Monthly inference cost
$180
Docs indexed
1,247 pages
User satisfaction (thumbs up)
81%
Background
DataPulse sells workflow analytics to ops teams. Their docs site had 1,200+ pages across guides, API references, and changelog entries. Search was Algolia keyword-only — fine for people who knew the product vocabulary, useless for new users asking "how do I connect Salesforce?"
Support was drowning in repeat questions. The docs team couldn't write fast enough. Leadership wanted an AI assistant but had no ML headcount.
Constraints
- Timeline: Ship in three weeks before a product launch
- Team: One backend engineer (part-time), one technical writer for eval questions
- Budget: Under $300/month inference
- Risk: Answers had to cite sources — no hallucinated API endpoints
Approach
Week 1 — Index and eval
The engineer exported docs from their headless CMS as Markdown, chunked on ## headings with 10% overlap, and embedded with text-embedding-3-small into pgvector on their existing RDS instance.
In parallel, the writer built a 50-question eval set from real support tickets: question, expected doc URL, difficulty tag.
Week 2 — Retrieval + generation
They built a /api/docs-chat endpoint:
- Embed the user question
- Run hybrid search — BM25 on title + heading, dense search on chunk body, reciprocal rank fusion to merge
- Rerank top 10 → top 5 with a lightweight cross-encoder (added in week 4 retrospectively; week 2 used top-k only)
- Stream a cited answer from Claude Sonnet with a strict system prompt
The chat widget was a floating panel on every docs page, built in Next.js and deployed on Vercel alongside the docs site.
Week 3 — Launch and iterate
Soft launch to 10% of docs traffic with a feedback thumbs-up/down on every answer. They watched three metrics daily:
- Retrieval recall on the eval set (did the right chunk appear in top-5?)
- Thumbs-up rate
- Support ticket volume for tagged topics
Results
After full rollout at week 5:
| Metric | Before | After |
|---|---|---|
| Repeat "how do I…" tickets | 142/week | 94/week |
| Median time-to-answer (user) | 4–8 min (search + read) | 2.1s |
| Docs search bounce rate | 38% | 27% |
The content flywheel was unexpected: logged unanswered questions became the docs team's priority queue. They shipped 12 new articles in month two directly from chatbot gaps.
What didn't work
- Fixed 512-token chunks in the v0 prototype — broke API reference pages where one endpoint spanned multiple chunks. Heading-based chunking fixed it.
- No refusal path in the first prompt version — the bot guessed on billing questions outside the docs corpus. Adding "I don't see that in our docs" + support link cut complaints.
- Embedding the full conversation history — ballooned latency. They kept only the last user message for retrieval and passed the last 3 turns for generation context.
Stack cost breakdown
| Item | Monthly cost |
|---|---|
| Embeddings (re-index on deploy) | ~$12 |
| Claude Sonnet inference (~8K queries) | ~$145 |
| pgvector on existing RDS | $0 incremental |
| Vercel (widget + API) | ~$23 |
| Total | ~$180 |
Takeaway
A docs chatbot is a retrieval problem dressed up as a chatbot. DataPulse's win wasn't picking the best model — it was building an eval set, fixing chunking, and logging failures so the docs team could close the loop.
Lessons learned
- 01
Hybrid search (BM25 + embeddings) beat embeddings-only on their API-reference-heavy docs — 18% higher retrieval recall on the eval set.
- 02
Chunking on Markdown headings, not fixed token windows, eliminated half the 'right answer, wrong section' failures.
- 03
A strict 'cite or refuse' system prompt reduced hallucination reports to near zero, but required a fallback link to human support for edge cases.
- 04
Logging every Q&A pair to Postgres let them find the top 20 unanswered questions and write new docs — a content flywheel they didn't plan for.
- 05
They shipped v1 without reranking; adding a lightweight cross-encoder reranker in week 4 improved answer quality more than switching models.
Frequently asked questions
Related on AIKnowHub
Workflow
Build an AI Docs Chatbot
A RAG-powered chatbot that answers questions from your documentation with citations — the kind every SaaS site needs in 2026.
Concept
Chunking Strategies for RAG
Chunking is the highest-leverage decision in any RAG system. Get it wrong and no embedding model or vector DB will save you. Here's how to chunk by document type.
Concept
RAG Explained
Retrieval-Augmented Generation is how you give LLMs access to your own data without retraining them. Here's how it actually works.
Interview
How do you handle PII in a RAG pipeline without breaking retrieval quality?
Tests privacy-aware AI design — a gap between demo RAG and enterprise deployment.
Interview
Walk me through designing a production RAG system for a docs chatbot.
System-design-style question that's now standard in AI engineering interviews.
Tool Guide
ChatGPT Guide
The most popular general-purpose AI chatbot. Strong all-rounder with a huge ecosystem of plugins, custom GPTs, and integrations.