LLM Cost Optimization

The real problem

Your LLM bill isn't high because models are expensive. It's high because every request hits the most expensive model with the longest possible context. That's a design choice, not a law of physics.

Most production workloads are a mix: 70% classification, extraction, and formatting; 20% moderate reasoning; 10% genuinely hard. If you're sending all of it to Claude Opus or GPT-5 at full context, you're lighting money on fire.

The optimization stack (in order of leverage)

1. Model routing — the 10x move

Route by task difficulty, not by user tier.

Incoming request
  → cheap classifier (Gemini Flash, Haiku, GPT-5 nano)
  → easy? → cheap model handles it
  → hard? → escalate to frontier
  → ambiguous? → cheap model tries first, escalate on low confidence

What counts as "easy":

• Classification and routing
• Structured extraction (name, date, category)
• Summarization under 2K output tokens
• Format conversion (JSON → markdown, etc.)

What needs frontier:

• Multi-step reasoning with tool use
• Code generation with test feedback loops
• Nuanced judgment calls (legal, medical, financial)

A well-tuned router cuts frontier-model calls by 60–80%. That's not a marginal saving — it's a different cost structure.

2. Prompt caching — free money you're leaving on the table

If your system prompt, tool definitions, or RAG context prefix is identical across calls, prompt caching stores it server-side after the first request. Subsequent calls pay 75–90% less on that prefix.

This matters most when:

• Your system prompt is 2K+ tokens (common in agent setups)
• You run RAG with a fixed retrieval template
• You make thousands of similar calls per day

Enable it. Measure the hit rate. If it's below 50%, your prompts aren't structured for cache reuse — fix that before switching models.

3. Context compression — shorter is cheaper

Every token in the context window costs money. Most agent loops accumulate garbage:

• Full chat history when only the last 3 turns matter
• Tool outputs from 5 steps ago that are already reflected in the answer
• Retrieved chunks that weren't used in the previous response

Practical fixes:

• Summarize conversation history every N turns (keep a rolling summary + recent messages)
• Cap RAG retrieval at 5–8 chunks, not 20
• Strip tool outputs from context once their result is incorporated
• Use a smaller model to compress before sending to the frontier model

Context engineering is cost engineering. See our Context Engineering Explained guide for the full playbook.

4. Batch APIs — half price for offline work

OpenAI and Anthropic both offer batch endpoints at ~50% discount. The tradeoff: results return within 24 hours, not seconds.

Perfect for:

• Eval suite runs
• Nightly document processing
• Bulk summarization and tagging
• Embedding generation for index updates

If you're running these synchronously today, you're paying double for no reason.

5. Output control — stop the runaway generation

Unbounded max_tokens is a budget leak. Set explicit limits:

• Classification: 50–100 tokens
• Extraction: 200–500 tokens
• Summarization: 500–1500 tokens
• Only open-ended generation gets 4K+

Combine with structured outputs (JSON mode, tool schemas) to prevent the model from rambling. A 200-token structured response beats a 2K-token prose answer every time — for cost and reliability.

6. Fine-tune small models for narrow tasks

If you're paying a frontier model to do the same narrow task 10,000 times/day (support ticket classification, intent routing, format conversion), fine-tune a small model and cut per-call cost by 90%+.

Fine-tuning shifts behavior, not knowledge — so pair it with RAG for facts. But for stable, repetitive tasks, it's the cheapest long-term play. See Fine-Tuning Explained for when it actually makes sense.

Metrics that matter

Stop tracking raw token spend. Track these instead:

The anti-patterns (what wastes money)

1. One model for everything — frontier on every call because "quality matters." It doesn't for 70% of your traffic.
2. Stuffing full docs into context — use RAG retrieval, not paste-the-whole-PDF. See RAG Explained.
3. Ignoring caching — paying full input price on identical prefixes thousands of times daily.
4. No evals before optimizing — you cut cost and quality together, then can't tell which broke.
5. Self-hosting prematurely — GPU ops, cold starts, and model updates cost more than API routing until you're at serious scale.

A realistic savings timeline

These aren't theoretical. They're what teams actually see when they stop treating every LLM call like a research project.

The honest take

Cost optimization isn't about using worse models. It's about using the right model for the right task, with the right amount of context, cached where possible. The teams that do this well spend 1/5th what everyone else spends — and their apps are faster because shorter contexts mean lower latency too.

Start with routing and caching. Measure with evals. Optimize the boring 70% of your traffic first. The hard 10% can keep the frontier model.