LLM Observability in Production

The visibility problem

Traditional software fails loudly — exceptions, stack traces, HTTP 500s. LLM applications fail quietly:

• The answer is wrong but well-formed.
• The response is slow but doesn't timeout.
• The cost is 3x higher than expected but nothing errors.
• Retrieval returns irrelevant chunks and the LLM hallucinates confidently.

Your existing monitoring (uptime checks, error rates, CPU graphs) won't catch any of these. You need observability designed for probabilistic systems.

The three pillars for LLMs

Logs — what happened on this request

Every LLM call should produce a structured log entry:

{
  "trace_id": "abc-123",
  "span": "llm.generate",
  "model": "claude-sonnet-4-20250514",
  "input_tokens": 1842,
  "output_tokens": 387,
  "latency_ms": 2340,
  "cost_usd": 0.0089,
  "status": "success",
  "user_id": "usr_hashed_abc",
  "feature": "docs-chatbot"
}

Store the full prompt and completion too — but redact PII first.

Traces — how the request flowed

A single user question in a RAG system touches multiple steps:

User query
  ├── span: embed_query (12ms)
  ├── span: vector_search (45ms)
  ├── span: bm25_search (8ms)
  ├── span: rerank (32ms)
  ├── span: llm_generate (2340ms)    ← the bottleneck
  └── span: format_response (2ms)

Distributed tracing links these spans into a single trace. When a user reports a bad answer, you open the trace and see exactly which step failed — bad retrieval, wrong model, truncated context.

Metrics — aggregates over time

Metrics tell you that something is wrong. Traces tell you what.

The production stack

Option 1: Managed LLM observability

Quickest start: Helicone as a proxy — change your API base URL, get tracing immediately. No SDK integration.

Most control: Langfuse self-hosted — full tracing, prompt versioning, eval integration, no vendor lock-in.

Option 2: OpenTelemetry

OpenTelemetry (OTel) is the open standard for distributed tracing. LLM semantic conventions are maturing:

from opentelemetry import trace

tracer = trace.get_tracer("my-rag-app")

with tracer.start_as_current_span("llm.generate") as span:
    span.set_attribute("llm.model", "claude-sonnet-4-20250514")
    span.set_attribute("llm.input_tokens", 1842)
    span.set_attribute("llm.output_tokens", 387)
    response = await llm.generate(prompt)

Export to Jaeger, Grafana Tempo, Datadog, or any OTel-compatible backend. Use OTel when you already have an observability stack and want LLM spans alongside your existing infrastructure traces.

Option 3: Roll your own

For early-stage projects, structured logging to your existing log aggregator (Datadog, Grafana Loki, CloudWatch) works:

logger.info("llm_call", extra={
    "model": model,
    "input_tokens": usage.input_tokens,
    "output_tokens": usage.output_tokens,
    "latency_ms": elapsed,
    "cost_usd": calculate_cost(usage),
    "trace_id": trace_id,
})

Upgrade to a dedicated LLM observability tool when you need trace visualization, prompt comparison, or eval integration.

What to instrument

Every LLM call

• Model name and version
• Input and output token counts
• Latency (time to first token + total)
• Cost (calculated from token counts × price)
• Status (success, timeout, rate-limited, content-filtered)
• Full prompt and completion (redacted)

Every retrieval step

• Query text
• Number of candidates retrieved
• Top-k chunk IDs and scores
• Retrieval latency
• Which retriever (dense, sparse, hybrid)

Every tool call (agents)

• Tool name and input parameters
• Tool output (truncated if large)
• Latency and success/failure
• See AI Agents Explained for agent-specific patterns

User feedback

• Thumbs up/down on responses
• Support ticket links
• Explicit corrections ("that's wrong, the answer is X")

Cost observability

LLM cost is the metric teams ignore until the invoice arrives.

Track per request:

cost = (input_tokens × input_price) + (output_tokens × output_price)

Aggregate by:

• Feature / endpoint
• User / team / tier
• Model
• Time period (hourly, daily, monthly)

Set guardrails:

• Per-user daily budget
• Per-feature monthly cap
• Alert when daily spend exceeds 2x the 7-day average

A single runaway agent loop can burn hundreds of dollars in minutes. Cost alerts are as important as error alerts.

Connecting observability to evals

Observability finds problems. Evals prevent them from recurring.

The production loop:

1. User reports bad answer → find the trace.
2. Inspect the trace → identify the failure (bad retrieval, wrong prompt, model hallucination).
3. Add to eval set → create a test case from the real failure.
4. Fix and re-run evals → verify the fix doesn't regress other cases.
5. Deploy → monitor metrics for improvement.

Also run automated sampling: every day, pull 50 random production traces, score them with an LLM-as-judge, and alert if the weekly average drops. This catches drift that users don't report.

Dashboards that matter

Skip the vanity metrics. Build dashboards for:

1. Latency waterfall — p50/p99 per step (retrieval, rerank, generate). Shows where time goes.
2. Cost breakdown — spend by feature and model. Shows where money goes.
3. Quality trend — automated eval score over time. Shows if quality is drifting.
4. Error and timeout rate — by model and endpoint. Shows reliability.
5. Feedback ratio — negative feedback per 100 requests. Shows user satisfaction.

Common failure modes

• Logging prompts with PII — user emails, medical data, financial info in trace storage. Redact at ingestion.
• No trace IDs across steps — you can see the LLM call but can't connect it to the retrieval that fed it. Use a single trace ID per user request.
• Alerting only on errors — LLM systems degrade gracefully. A model switch that doubles latency won't error — it'll just be slow. Alert on latency and cost too.
• Observability added after launch — the first month of production is when you need visibility most. Instrument before you ship.
• Traces without retention policy — prompt/completion storage grows fast. Set TTL (30–90 days) and archive to cold storage.

The minimum viable observability stack

For a team shipping their first LLM feature:

1. Structured logging on every LLM call (model, tokens, latency, cost).
2. A trace ID propagated through retrieval → generation → response.
3. A daily cost report by feature.
4. User feedback collection (thumbs up/down).
5. Weekly eval run on 50 sampled production traces.

That's enough to debug issues, control costs, and catch drift. Add Langfuse/Helicone/Braintrust when you need trace visualization, prompt versioning, or automated eval pipelines.

Observability isn't optional for production AI. It's the difference between a demo and a product. See Skills for an AI Engineer for where this fits in the broader production skill set.