RAG Stack Under $100/mo

The stack

Layer 1 — Ingestion

Web docs: Firecrawl URL to markdown. Schedule weekly re-crawl for stale pages.

Git/markdown repo: Direct chunk from files — no crawl needed.

PDFs: pymupdf or unstructured.io locally.

→ Chunking Strategies

Layer 2 — Embeddings

Default: text-embedding-3-small at 1536 dimensions.

Cost math: 1,200 doc pages at ~500 tokens/chunk, 3 chunks/page = 1.8M tokens. One-time embed under $0.50. Re-embed on change only.

Layer 3 — pgvector on Supabase

create extension vector;
create table chunks (
  id bigserial primary key,
  content text,
  embedding vector(1536),
  metadata jsonb
);
create index on chunks using hnsw (embedding vector_cosine_ops);

Store source_url, title, section in metadata for citations.

Upgrade to Qdrant or Pinecone when evals show retrieval latency problems at scale.

Layer 4 — Retrieval + generation

query
  → embed query
  → top-k similarity (k=8)
  → optional rerank (Cohere rerank or LLM pick top 3)
  → prompt with chunks + cite instructions
  → Haiku for default, Sonnet if confidence low

→ Reranking and Hybrid Search

Layer 5 — Observability

Log every query in Langfuse:

• Retrieved chunk IDs
• Answer latency
• User thumbs down (feeds eval set)

The RAG docs case study hit $180/mo at production traffic — still under $100 for MVP volumes.

Cost control levers

1. Cache embeddings — never re-embed unchanged docs
2. Haiku default — Sonnet only on escalation
3. Smaller k — start k=5, increase only if evals fail
4. Query routing — FAQ keyword match before vector search

→ LLM Cost Optimization

Build path

Follow Build an AI Docs Chatbot — it maps directly to this stack.

Upgrade path