Directory · RAG
LlamaIndex
Open-source framework for connecting LLMs to data with indexes and query engines; LlamaCloud adds managed parsing.
Best for
Building RAG pipelines and document agents over private data
More rag tools
Crawl4AI
RAG · Crawl4AI (open source)
Open-source web crawler built for LLM pipelines — async Playwright, clean markdown output, and LLM-friendly extraction.
Best for
Self-hosted RAG ingestion, research automations, replacing brittle scrapers
Pros
- Fully async and scriptable
- No SaaS dependency for sensitive crawls
- Active development in 2026
Cons
- You operate Playwright and browsers
- Slower to ship than Firecrawl API
- Complex sites need custom extract rules
Alternatives
Firecrawl
RAG · Firecrawl · United States
Web-to-markdown API for AI pipelines — crawl sites, extract clean content, and feed RAG indexes without brittle scrapers.
Best for
RAG ingestion, competitive intel, docs mirroring, research automations
Pros
- Clean markdown output (not raw HTML soup)
- Handles JS-rendered pages
- Self-host option for sensitive crawls
Cons
- Rate limits on hosted tier
- Complex sites still need custom rules
- Crawling ethics and robots.txt are on you
Alternatives
Haystack
RAG · deepset · Germany
deepset's open-source Python framework for building RAG pipelines and search systems from composable components.
Best for
Production RAG pipelines with swappable retrievers, rankers, and generators