How to Observe LLM Apps with OpenTelemetry

What to observe in an LLM application

LLM applications have signals no traditional service has, and they map directly to cost and user experience: prompt and completion tokens per request (which drive the bill), model latency and time-to-first-token, error and rate-limit responses, and, for retrieval-augmented apps, the retrieval step and how relevant its results were. For agents, each tool call and reasoning step is its own span, so a runaway loop is visible as a trace that will not end.

The two questions this telemetry has to answer continuously are 'what is each request costing us' and 'why is this response slow or wrong', and both are answered at the level of the individual model call, which is why per-request tracing with token and model attributes is the foundation.

How OpenTelemetry collects it

OpenTelemetry has GenAI semantic conventions for LLM spans, and instrumentation libraries such as OpenLLMetry and OpenInference wrap the common SDKs, OpenAI, vLLM, Anthropic, and orchestration frameworks like LangChain and LlamaIndex, so every model call emits a span carrying the model, token counts and latency as gen_ai.* attributes, exported over OTLP. For agents, the framework instrumentation nests tool and step spans under the request so the whole reasoning chain is one trace.

Two collection decisions matter here that do not arise elsewhere. First, prompts and completions can contain personal or sensitive data, so decide deliberately whether to capture prompt and response bodies, and redact or sample them if you do. Second, high-volume LLM traffic is expensive to store in full, so sample traces while keeping the token and cost metrics complete, since the aggregate cost view must not be sampled away.

Key metrics to watch

The signals that explain most LLM application incidents:

What to put on a dashboard

An LLM dashboard should keep cost and experience side by side. Row one is cost: tokens per request by model and total spend over time, with the per-request token distribution so a creeping prompt size is caught before the bill is. Row two is experience: model latency and time-to-first-token, and the error and rate-limit rate. Row three, for RAG and agents: retrieval latency and result counts, and, for agents, steps per request so a runaway loop shows up as an outlier.

# tokens per request by model (rising = prompt bloat or bigger outputs)
gen_ai.usage.input_tokens + gen_ai.usage.output_tokens  by (gen_ai.request.model)

# estimated cost per model (tokens x price)
sum(gen_ai.usage.output_tokens * price_out + gen_ai.usage.input_tokens * price_in) by (gen_ai.request.model)

# rate-limit (429) rate
rate(gen_ai_errors{type="rate_limit"}[5m]) by (gen_ai.request.model)

Reading the signals: common failure modes

Cost spike

Spend jumps without a matching jump in traffic, because tokens per request climbed, a prompt template grew, context is being stuffed, or an agent is looping and re-calling the model. The per-request token distribution reveals it, and the trace of a single expensive request shows exactly which step ballooned. Cap context, trim prompts, or bound agent iterations.

Rate-limit errors

429 responses rise as you approach the provider's tokens-per-minute or requests-per-minute limit, and user requests start failing or retrying. The rate-limit rate by model is the signal. Spread load, request a higher quota, add backoff, or route overflow to a secondary model.

Latency from large context

Model latency and time-to-first-token climb because prompts carry a large context window or outputs are long. The duration metric rises with token count. Reduce retrieved context, cap output length, or stream so the user sees tokens sooner.

Poor retrieval in RAG

Answers degrade even though the model is healthy, because the retrieval step returns too few or irrelevant chunks. Retrieval span latency and result count, alongside the retrieved content, expose it. The fix is in the index, chunking or query, not the model.

Example alert conditions

Starting thresholds to tune:

From monitoring to autonomous resolution

Ops Singularity's AI/ML and Agent Ops pillar operates LLM apps and agents in production, ingesting this OpenTelemetry data, watching cost, latency and quality, and taking governed action when an agent or model degrades. Explore AI/ML and Agent Ops and TelemetryOps.