Skip to content

Tool Guide · Infrastructure

LiteLLM Guide

The definitive guide to LiteLLM for multi-model routing — OpenAI-compatible proxy, fallbacks, cost tracking, and self-host setup for production AI apps.

9 min readOpen-source self-host (free) + LiteLLM Cloud optionalUpdated Jul 2026
LiteLLMAPIRoutingOpen SourceCost
Visit official site
Edited by The AIKnowHub team · Editorial team

Key takeaways

  • 1OpenAI-compatible /v1/chat/completions proxy
  • 2100+ provider support
  • 3Fallbacks and load balancing
  • 4Per-key budgets and rate limits
  • 5Cost logging and admin UI
  • 6Drop-in OpenAI SDK replacement
  • 7Self-host via Docker

Best for

  • Multi-provider model routing
  • Automatic failover on outages
  • Centralized API key management
  • Cost attribution per team or feature
  • n8n/Pipedream HTTP steps calling one endpoint

Not for

  • Single-provider apps with no failover needs
  • Notebook one-off scripts
  • Teams allergic to running a proxy service

What it is

LiteLLM is an open-source LLM gateway. Point your app at one OpenAI-compatible URL; LiteLLM routes to Anthropic, OpenAI, Google, Groq, local models, and more.

Anchors the automation engineer stack alongside n8n and Langfuse.

Minimal config

model_list:
  - model_name: fast
    litellm_params:
      model: anthropic/claude-haiku-4-5
  - model_name: smart
    litellm_params:
      model: anthropic/claude-sonnet-4-5

router_settings:
  fallbacks: [{"fast": ["gpt-4.1-mini"]}]

Client code:

from openai import OpenAI
client = OpenAI(base_url="http://litellm:4000", api_key="sk-...")
client.chat.completions.create(model="fast", messages=[...])

Production patterns

PatternConfig
Classify cheap, draft qualityTwo model_names
Anthropic down → OpenAIfallbacks array
Per-customer budgetvirtual keys in admin UI
Dev vs prod keysseparate key aliases

Observability

Pair with Langfuse — LiteLLM supports callback hooks for traces and cost.

The email assistant case study used LiteLLM to A/B Haiku vs Sonnet for classification.

Self-host

Docker on Fly.io, Railway, or internal K8s. Budget 1-2 hours for first deploy with Postgres for spend logs.

When to skip

Single provider, low volume, no failover requirement — direct SDK is fine until pain appears.

Related

Pros and cons

Pros

  • One integration for every major model
  • Fallbacks prevent user-facing outages
  • Budget caps stop runaway spend
  • Works with existing OpenAI client code

Cons

  • Another service to operate
  • Provider edge cases still leak through
  • Config YAML grows complex at scale

Real workflows using this tool

Alternatives

PortkeyHeliconeDirect provider SDKs

Frequently asked questions

LiteLLM for routing and fallbacks. Helicone for fastest cost-tracking proxy. Many teams use both — LiteLLM routes, Helicone logs.

Watch

Hand-picked videos from official + trusted channels. Opens in a new tab.

Related on AIKnowHub