---
title: "How to monitor LLM apps in production"
excerpt: "A practitioner's guide to tracing, cost and latency metrics, online quality checks, drift detection, and feedback loops for production LLM systems."
date: "2026-07-09"
lastModified: "2026-07-09"
summary: "LLM apps fail differently than normal services. This guide covers the tracing, metrics, quality sampling, and feedback loops you need to run them safely in production."
keyTakeaways:
  - "Trace the whole request chain as spans, not just the final generation call, so you can see where quality and latency actually go wrong."
  - "Track three metric families: cost and tokens, latency including time-to-first-token, and quality sampled online with an LLM judge."
  - "Close the loop by feeding failing traces and user feedback back into your eval golden set, so monitoring drives real improvement."
author: "Teo Deleanu"
authorAvatar: "/team/teo.jpg"
tags: ["Observability", "Monitoring", "AI Engineering", "Production AI"]
keywords:
  - "how to monitor llm apps in production"
  - "llm observability"
  - "llm tracing"
  - "llm cost monitoring"
  - "llm drift detection"
  - "llm as a judge monitoring"
---

Most teams instrument an LLM app the way they instrument a REST service: request count, p95 latency, error rate, a dashboard, done. That catches crashes and timeouts. It tells you almost nothing about whether your app is actually doing its job. An LLM endpoint can return 200 OK, in 400ms, with a fluent, confident, completely wrong answer. Standard monitoring is blind to exactly the failure mode that matters most.

The reason is non-determinism. The same prompt can produce different outputs across calls, model versions, and provider-side changes you never get told about. And "correct" is subjective and task-specific. There is no exception thrown when the model invents a policy or ignores a retrieved document. So monitoring an LLM app is really two jobs stacked together: the classic operational one (is it up, fast, and affordable) and a quality one (is it still good). You need both, and they use different tools.

## Trace the full request chain

Start with tracing, because in a real LLM app the interesting failures live between the components, not inside the final model call. A typical request fans out: embed the query, retrieve candidates, rerank, assemble a prompt, call the model, maybe call a tool, maybe loop. If you only log the final generation, a bad answer is unattributable. Was retrieval empty? Did the reranker bury the right chunk? Was the prompt truncated? You can't tell.

Model each step as a span in a single trace, OpenTelemetry-style. Parent span for the request, child spans for retrieval, rerank, and generation, each carrying its own attributes: the query, how many documents came back, token counts, latency, the model name and version. GenAI semantic conventions in OpenTelemetry now cover common LLM attributes, so you don't have to invent a schema from scratch, and most LLM observability vendors ingest OTel spans directly. If your app is built on an orchestration framework, its tracing story matters here — [LangChain vs LlamaIndex](/compare/langchain-vs-llamaindex) differ in how much instrumentation they emit out of the box.

```python
from opentelemetry import trace

tracer = trace.get_tracer("rag.app")

def answer(query: str) -> str:
    with tracer.start_as_current_span("rag.request") as root:
        root.set_attribute("input.query", query)

        with tracer.start_as_current_span("retrieval") as s:
            docs = retriever.search(query, k=8)
            s.set_attribute("retrieval.k", 8)
            s.set_attribute("retrieval.hits", len(docs))

        with tracer.start_as_current_span("generation") as s:
            s.set_attribute("gen.model", "gpt-4o-mini")
            prompt = build_prompt(query, docs)
            resp = client.chat(prompt)
            usage = resp.usage
            s.set_attribute("gen.input_tokens", usage.input_tokens)
            s.set_attribute("gen.output_tokens", usage.output_tokens)
            root.set_attribute("output.text", resp.text)
        return resp.text
```

The rule of thumb: if a step can degrade the answer, it deserves a span. That instrumentation is also what makes everything below possible, because your metrics and quality checks read from the same traces.

## The three metric families

Once traces carry token counts, latency, and model metadata, roll them up into three families.

**Cost and tokens.** LLM cost is driven by input and output tokens, and it is easy for costs to climb quietly. A prompt template that grows, a retriever that returns more chunks, an agent that loops an extra turn. Track tokens and estimated cost per request, per feature, and per user or tenant. Watch output tokens specifically, since they are usually priced higher and are where runaway generations show up. Set alerts on cost-per-request trend, not just total spend, so a regression stands out before the monthly bill does.

**Latency.** Wall-clock latency matters, but for anything user-facing, time-to-first-token (TTFT) matters more, because streaming apps feel fast when the first token lands quickly even if total generation is slow. Track TTFT and total latency separately, and break latency down by span so you know whether retrieval, reranking, or the model is the bottleneck. If you self-host, the serving layer is often where you tune this; our guide on [how to deploy vLLM in production](/guides/how-to-deploy-vllm-in-production) goes into batching and TTFT there.

**Quality.** This is the family classic monitoring skips, and it needs its own approach.

## Measuring quality online

You cannot wait for offline evals to tell you production quality is slipping. You need a signal on live traffic. The practical method is to sample a fraction of real traces and score them automatically, most commonly with [LLM-as-a-judge](/glossary/llm-as-a-judge): a second model call that rates the response against criteria you define, such as faithfulness to retrieved context, relevance to the question, or adherence to format.

Sample rather than score everything. Judging every request doubles your model spend and adds latency; scoring 1 to 5 percent of traffic is usually enough to see trends and catch regressions. Run the judge asynchronously, off the request path, so it never slows the user's response. For [retrieval-augmented generation](/glossary/retrieval-augmented-generation), a faithfulness judge that checks whether the answer is grounded in the retrieved chunks is one of the highest-value online checks you can run, because it catches [hallucination](/glossary/hallucination) directly.

Be honest about the tradeoff: an LLM judge is itself an LLM, so it is noisy and can be biased toward verbose or confident answers. Treat judge scores as a directional signal on aggregate trends, not as ground truth on individual requests. Calibrate the judge against a small human-labeled set before you trust it, and revisit that calibration when you change judge models. Judge design is close enough to offline evaluation that it is worth reading [how to evaluate LLMs](/guides/how-to-evaluate-llms) and the [LLM evals](/glossary/llm-evals) glossary entry before you write your rubric.

## Detecting drift

Drift is what makes an app that shipped fine slowly get worse, and it comes from both ends.

**Input drift.** The distribution of what users send you shifts. New topics, longer queries, a new language, a different intent mix than you designed for. Track cheap proxies: query length, language, embedding-cluster distribution, retrieval hit rates, out-of-scope rates. A rise in empty retrievals or a new dominant cluster is often the first sign that reality has moved away from your test set.

**Output drift.** Quality degrades even with stable inputs, usually because a provider silently updated a model, or your own prompt or retrieval index changed. Your online judge scores and faithfulness rates are the drift detector here. A slow decline in average faithfulness, or a rise in format violations, is your early warning. This is also the strongest argument for pinning model versions where your provider allows it, so a silent upgrade shows up as an intentional change you can test rather than a mystery regression.

## Closing the feedback loop

Monitoring that only produces dashboards is half a system. The point of catching bad traces is to make the app better, and that means the loop has to close back into your evals.

Two inputs feed the loop. First, explicit and implicit user feedback: thumbs up and down, edits, retries, abandonment, escalations to a human. Capture these against the trace ID so a rating is tied to the exact request, retrieval set, and model version that produced it. Second, low-scoring traces from your online judge. Both are candidates for your golden set.

The workflow that pays off: route failing traces and negative feedback into a review queue, have a human confirm which are genuine failures, then add the confirmed cases to your offline eval set as new golden examples. Now every regression you caught in production becomes a permanent test. The next prompt change, model swap, or retrieval tweak gets scored against real failures you have already seen, not just the happy-path cases you imagined at design time. Over time your eval set stops being a synthetic artifact and becomes a distilled record of how your app actually breaks.

That is the whole loop: trace everything, measure cost, latency, and sampled quality, watch both ends for drift, and feed real failures back into evals. Standard observability keeps the service up. This keeps it good, which for an LLM app is the harder and more important half.