Local Models vs API Models

The short version

This isn't a religious war. It's a routing problem. The teams that win pick per task, not per team ideology.

Local models — privacy and predictable economics

Ollama is the default local runtime in 2026. Pull a model, get an OpenAI-compatible endpoint on localhost:11434, wire your app.

Where local wins:

• Privacy — prompts and completions stay on your machine or VPC. No subprocessor review for every new vendor.
• Marginal cost — after hardware, each additional token costs electricity, not dollars.
• Offline — works on a plane, in a lab, in an air-gapped datacenter.
• Model freedom — swap Qwen, Llama, DeepSeek, Mistral without renegotiating contracts.

Where local hurts:

• Quality ceiling — open weights are competitive on benchmarks but still trail Claude and GPT on long agent loops and judgment calls.
• Hardware — CPU inference is painfully slow. You want a GPU (NVIDIA) or Apple Silicon with enough unified memory.
• Ops — you own driver updates, OOM crashes, model versioning, and scaling. That's real work.
• Latency variance — a loaded GPU serving one user is fast; sharing across ten concurrent agent sessions is not.

Best local workloads: classification, PII detection, structured extraction, internal RAG on confidential docs, dev-time prototyping.

API models — quality and zero ops

OpenAI and Anthropic (plus Gemini, Mistral hosted, etc.) sell inference as a service. You send tokens, they send completions.

Where APIs win:

• Quality — frontier models still lead on reasoning, coding agents, and multi-step tool use.
• Time to ship — API key + SDK. No GPU procurement cycle.
• Reliability — someone else handles uptime, scaling, and security patches.
• Features — structured outputs, prompt caching, batch APIs, evals tooling — mature and documented.

Where APIs hurt:

• Cost at scale — linear with usage. Teams without routing discipline get surprise bills.
• Data governance — even with zero-retention and BAA options, legal may block third-party inference.
• Vendor lock-in — switching providers means retesting prompts, evals, and latency profiles.

Best API workloads: customer-facing chat, coding agents (Claude Code, Cursor), complex reasoning, anything where quality regressions cost revenue.

Hybrid — what production actually looks like

Most mature AI products in 2026 don't pick one lane. They route.

A typical hybrid stack:

1. Ingress — classify the request (local small model or rules).
2. Sensitive path — PII-heavy steps run locally (redact, embed, retrieve).
3. Generation path — scrubbed context goes to a frontier API for the hard answer.
4. Egress — validate output (guardrails), log for observability.

Why hybrid works: you get privacy where contracts demand it and quality where users notice it.

Why hybrid hurts: two inference paths, two failure modes, routing logic to maintain. Don't build this on day one unless compliance requires it.

The numbers that matter

Cost break-even (rough)

Electricity + a $2,000 GPU amortized over 24 months ≈ $100–150/mo in hardware cost. Compare that to $0.15–$3/1M tokens on APIs depending on model tier. Run your actual prompt distribution through a spreadsheet — marketing "local is free" ignores amortization.

Latency

• Local on M3 Max / RTX 4090: 30–80 tok/s for 7B–14B models. First token often 100–400ms.
• API (Claude/GPT): 200–800ms first token, consistent regardless of your hardware.
• Groq (hosted open weights): can beat local on latency without owning a GPU.

Latency-sensitive UX (live autocomplete, voice) often stays on API or specialized fast hosts even when batch work is local.

Quality

Run your eval set on both paths. Seriously.

A 14B local model may score 90% on your classification tasks and 60% on your agent tasks. That's a routing signal, not a verdict. See Evals Explained for how to build the test set.

Decision tree

Data cannot leave your network / air-gapped?
  └─ Yes → Local (Ollama)

Shipping MVP in < 2 weeks, no compliance blocker?
  └─ Yes → API

> 5M tokens/month, mostly routine classification/extraction?
  └─ Yes → Local or hosted open-weights

Need frontier quality AND strict PII handling?
  └─ Hybrid (local preprocess → API generate)

No GPU and no budget for cloud GPU?
  └─ API (or Groq / Together for cheap open weights)

Common mistakes

1. Buying hardware before validating the use case. Run on API for two weeks, measure token volume and task types, then decide.
2. Running a 70B model on a laptop CPU. You'll conclude "local doesn't work" when the problem is hardware mismatch.
3. Sending everything to Opus. Local Haiku-class models handle routing; frontier models handle reasoning. See LLM Cost Optimization.
4. Ignoring ops. Local inference at 3am when the GPU driver crashes is your problem, not Ollama's.
5. Treating privacy as binary. Hybrid with redaction often satisfies legal without sacrificing quality on the generation step.

The practical answer

• Solo dev / side project: Ollama on your Mac. Free, fast enough, private.
• Startup pre-PMF: API. Ship features, not infrastructure.
• Enterprise with compliance: Local or hybrid from day one. Document the routing policy.
• High-volume SaaS: Hybrid or hosted open-weights. Route cheap tasks local, hard tasks frontier.

Pick the bottleneck: if privacy is the blocker, go local. If quality is the blocker, go API. If both are blockers, build hybrid — but earn the complexity by shipping something simpler first.