Which models should I start with?

General: Qwen3 14B or Llama 3.3 8B. Coding: Qwen3 Coder or DeepSeek Coder. Fast classification: Llama 3.2 3B or Gemma 3 4B. Pull with `ollama pull ` and benchmark on your actual tasks — see [Local vs API](/comparisons/local-models-vs-api).

Ollama or just use the API?

API for shipping fast and hard reasoning. Ollama for privacy, offline, and high-volume cheap tasks. Most production teams hybrid — see the full breakdown in [Local Models vs API](/comparisons/local-models-vs-api).

How do I add a chat UI?

Install Open WebUI (`docker run` with Ollama backend) for a ChatGPT-like interface. Or use Continue in VS Code, Aider in terminal, or n8n HTTP nodes pointing at localhost:11434.

Tool Guide · Local LLM

Ollama Guide

Run LLMs locally with Ollama — install, model selection, hardware requirements, OpenAI-compatible API, and pairing with Open WebUI for a private ChatGPT experience.

10 min readFree (open source) — you pay for hardware or cloud GPU timeUpdated Jun 2026

OllamaLocal LLMPrivacyOpen SourceSelf-HostGPU

Visit official site

Edited by The AIKnowHub team · Editorial team

Key takeaways

1One-command install on macOS, Linux, and Windows
2Pull and run models from a curated registry (Llama, Qwen, DeepSeek, Mistral, Gemma, and more)
3OpenAI-compatible REST API on port 11434 — drop into existing SDKs
4Model library with automatic quantization (Q4, Q8) for hardware-fit
5GPU acceleration on NVIDIA CUDA, Apple Metal, and AMD ROCm
6Multi-model support — switch models per request without restart
7Modelfiles — customize system prompts and parameters as reusable artifacts
8Docker image for server and container deployments
9Lightweight compared to full vLLM setup for single-user and small-team use
10Active model releases within days of upstream open-weights drops

Best for

Privacy-sensitive work — legal docs, internal code, PII-heavy drafts
Offline development and prototyping without API keys
High-volume classification and extraction where API cost adds up
Learning how LLMs behave before spending on APIs
Air-gapped or compliance-constrained environments
Pairing with Open WebUI, Continue, or Aider for a local dev stack

Not for

Production features requiring frontier reasoning quality
Teams without any GPU access (CPU-only is painfully slow)
Multi-tenant SaaS at scale without graduating to vLLM or hosted inference
Long agent loops where tool-use reliability is critical
I need GPT-4 quality on a 2019 Intel MacBook

What it is

Ollama is a local LLM runtime. Install it, run ollama pull llama3.3, and you have a model serving an OpenAI-compatible API on http://localhost:11434. No account. No API bill. Data stays on your machine.

It's the default on-ramp to local inference in 2026 — simpler than raw llama.cpp, lighter than a full vLLM cluster, and good enough for millions of developers who want privacy or cheap experimentation.

For the strategic "should I run local at all?" question, read Local Models vs API. This page is the how-to.

Quick start

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull qwen3:14b

# Chat in terminal
ollama run qwen3:14b

# API is now live at localhost:11434

Point any OpenAI SDK at base_url="http://localhost:11434/v1". Most tools (Continue, Aider, n8n HTTP nodes) accept this with minimal config.

Model selection (2026)

Model	Size	Best for
Llama 3.3 8B	Small	Fast general chat, low RAM
Qwen3 14B	Medium	Strong all-rounder — best default for most Macs
Qwen3 Coder	Medium	Code completion and review
DeepSeek R1 (distilled)	Medium	Reasoning on capable hardware
Llama 3.2 3B	Tiny	Classification, routing, tagging
Mistral Nemo 12B	Medium	European language tasks

Rule: start with Qwen3 14B or Llama 3.3 8B. Run your eval set. Upgrade size only if quality fails on real tasks, not benchmarks.

Pull multiple models — Ollama swaps per request. Use small models for routing, larger for generation (a simple hybrid pattern).

Hardware reality

Apple Silicon

M1/M2/M3 with 16GB+ unified memory runs 7B–14B models at usable speed (20–60 tok/s). M3 Max / M4 with 36GB+ handles 32B quantized models. This is why Ollama is popular on MacBooks — no NVIDIA required.

NVIDIA GPU

8GB VRAM → 7B–8B comfortably. 12GB → 14B quantized. 24GB → 32B. 48GB+ → 70B territory. CUDA acceleration is automatic when drivers are installed.

CPU only

Works for 3B models and batch scripts. Not for interactive chat or agent loops. If this is your only option, consider hosted open-weights via Cheapest AI APIs instead.

Cloud GPU

Rent a GPU instance (Lambda, RunPod, Vast.ai), install Ollama, expose API behind VPN. Common pattern for teams that need local-style privacy without buying hardware.

Open WebUI — the chat UI layer

Ollama is headless. Open WebUI adds a browser interface:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000, connect to Ollama backend, and you have private ChatGPT with model switching, conversation history, and RAG document upload.

Other UI options:

Continue — inline in VS Code / JetBrains
Aider — terminal pair programmer with Ollama backend
LibreChat — multi-provider UI including Ollama
Custom app — your frontend + Ollama API

Building with Ollama

RAG (retrieval-augmented generation)

Ollama handles generation only. You still need embeddings + vector store for RAG:

Embed documents (local embedding model or API).
Store in Qdrant, Chroma, or LanceDB.
Retrieve top-k chunks on query.
Send context + query to Ollama for the answer.

Full pattern: AI Docs Chatbot workflow — swap the LLM step to Ollama for a fully local stack.

Agents

Ollama can power simple agent loops, but open-weights models trail Claude/GPT on multi-step tool use. Fine for internal tools with 2–3 steps; risky for customer-facing autonomous agents. See AI Agents Explained.

For production agents, common pattern: Ollama for cheap preprocessing, frontier API for the reasoning loop.

Automation

n8n HTTP node → POST localhost:11434/api/chat inside a workflow. Great for self-hosted triage and classification without sending data to OpenAI.

Modelfiles

Customize a model's behavior permanently:

FROM qwen3:14b
PARAMETER temperature 0.2
SYSTEM "You extract JSON only. No markdown fences."

Save as Modelfile, then ollama create my-extractor -f Modelfile. Version control your Modelfiles — they're your local "system prompt as code."

Production considerations

Ollama is a dev and small-team tool by default. Scaling to production means:

Concern	Ollama default	Production path
Concurrent users	Limited	vLLM, TGI, or hosted provider
Failover	None	Load balancer + multiple instances
Observability	Basic logs	LLM Observability stack
Model updates	Manual pull	CI pipeline for model versioning
Security	Localhost	Reverse proxy, auth, network policies

Don't run Ollama exposed to the public internet without authentication. Put it behind VPN or an API gateway.

Ollama vs alternatives

Tool	When to pick
Ollama	Fastest setup, daily local use, OpenAI-compatible API
LM Studio	GUI-first, model browsing, no terminal comfort
vLLM	Production throughput, multi-GPU, OpenAI server at scale
Together / Fireworks	Open weights without owning hardware
Groq	Lowest latency hosted inference

Common mistakes

Pulling 70B on a 8GB GPU. Start small; scale up when evals demand it.
Expecting frontier quality. Local 14B ≠ Claude Opus. Route hard tasks to API.
No evals. "Feels worse" isn't actionable. Build a 20-prompt test set per Evals Explained.
Exposing port 11434 publicly. Localhost or VPN only.
Ignoring context limits. num_ctx defaults may truncate long RAG context — set explicitly in Modelfile.

The practical answer

Install Ollama today. Pull Qwen3 14B. Point Continue or Open WebUI at it. Run your internal docs through a local RAG experiment before committing to API spend.

If quality passes your eval bar for the task, keep it local. If not, you've lost an hour — not a quarter's infrastructure budget.

Pros

Free, open source, trivial setup vs raw llama.cpp or vLLM
Data never leaves your machine by default
Wide model catalog with sensible defaults
OpenAI-compatible API — minimal code changes to swap providers
Apple Silicon performance is genuinely good for 7B–14B models
No account, no API key, no rate limits

Cons

Quality ceiling below Claude/GPT on hard tasks
Large models (70B+) need serious VRAM or run slowly
No managed scaling, failover, or observability — you own ops
Slower than hosted APIs on underpowered hardware
Agent and tool-use loops less reliable than frontier models
Model updates are your responsibility

Real workflows using this tool

Prompt examples

Copy any of these, replace the placeholders, run.

OpenAI SDK pointed at Ollama

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required but ignored
)

response = client.chat.completions.create(
    model="qwen3:14b",
    messages=[{"role": "user", "content": "Classify this ticket: ..."}],
)

Modelfile for a custom assistant

FROM qwen3:14b

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

SYSTEM """
You are an internal support assistant. Answer only from provided context.
If unsure, say "I don't have enough information."
Never invent policy details.
"""

curl smoke test

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.3",
  "prompt": "Summarize RAG in two sentences.",
  "stream": false
}'

Alternatives

LM StudiovLLMllama.cppTogether / Fireworks (hosted open weights)Groq (fast hosted inference)

Frequently asked questions

Apple Silicon (M1+) with 16GB+ RAM handles 7B–14B models well. NVIDIA GPU with 8GB+ VRAM for similar. 70B models need 48GB+ unified memory or multi-GPU. CPU-only works for tiny models but expect 1–5 tokens/sec — fine for experiments, not production UX.

Watch

Hand-picked videos from official + trusted channels. Opens in a new tab.