Can I run RAG fully local?

Yes. Ollama embeddings (nomic-embed-text) + Chroma or pgvector + Open WebUI document upload. Quality depends on model and chunking.

When should I still use cloud APIs?

Complex coding agents, multi-step research, and customer-facing features where quality gaps hurt revenue. Many teams run local for internal, cloud for external.

Learn AI · Guides

Local Privacy AI Stack

Run models and chat entirely on your hardware — Ollama, Open WebUI, Continue, and local RAG without sending data to cloud APIs.

8 min readPublished Jun 2026Updated Jun 2026

StackLocalPrivacyOllama

Edited by The AIKnowHub team · Editorial team

Key takeaways

1Local stacks win on air-gapped, HIPAA-adjacent, and trade-secret code scenarios.
2You need a GPU — Apple Silicon works for 7B-13B; NVIDIA 24GB+ for larger.
3Open WebUI gives non-technical teammates a ChatGPT-like UI on local models.
4Continue in the IDE can point at Ollama instead of cloud APIs.
5Frontier model quality still requires API for hard tasks — hybrid is common.

The stack

Layer	Tool	Data leaves network?
Inference	Ollama	No
Chat UI	Open WebUI	No
IDE assist	Continue	No
Embeddings	Ollama nomic-embed-text	No
Vector store	Chroma or local pgvector	No
Docs ingestion	Local files / air-gapped git	No

Layer 1 — Ollama

ollama pull llama3.1:8b
ollama pull nomic-embed-text
ollama serve

Exposes OpenAI-compatible API at localhost:11434.

Model picks (2026):

Use	Model
General chat	llama3.1:8b or mistral
Code	deepseek-coder-v2 or qwen2.5-coder
Embeddings	nomic-embed-text

→ Local Models vs API

Layer 2 — Open WebUI

Browser chat for the team on your Ollama instance:

Document upload for RAG
Multi-user workspaces
Model switching per chat

Non-engineers get ChatGPT-like UX without cloud data transfer.

Layer 3 — Continue in IDE

Point Continue at http://localhost:11434/v1 for inline completions and chat in VS Code.

Developers stay in-editor; data stays local.

Layer 4 — Local RAG

Documents (git, PDF, markdown)
  → chunk locally
  → embed via Ollama nomic-embed-text
  → store in Chroma or pgvector
  → query via Open WebUI or custom app

No Firecrawl needed if docs are on disk. For internal wikis, export to markdown on schedule.

→ RAG Explained

Hybrid pattern (recommended)

Most privacy-conscious teams run:

Task	Where
Internal HR/policy Q&A	Local stack
Customer support bot	Cloud API + RAG
Code on secret repo	Local Continue
Marketing copy	Cloud (quality)

Draw the boundary by data classification, not ideology.

Hardware guide

Hardware	Comfortable models
M2 Pro 16GB	7B-8B Q4
M3 Max 36GB	13B Q4, multiple 8B
RTX 4090 24GB	13B-34B Q4
A100 40GB	70B Q4

Quantization (Q4_K_M) is standard — accept small quality trade for fit.

Limits

No computer-use browser agents at local model quality yet
Long context expensive on consumer GPU
Model updates are manual (ollama pull)
Team scale needs shared server, not laptops

When local is the wrong call

Customer-facing product where quality is revenue
You lack GPU hardware and won't buy it
Compliance allows BAA/DPA with Anthropic/OpenAI
You need frontier reasoning weekly

→ When NOT to Use AI

Related open-source

All in Top GitHub Repos and the directory.

Common misconceptions

The wrong-but-common takes worth correcting.

Myth

Local means free.

Reality

Hardware, electricity, and engineer time are real costs. License fees go away; ops do not.