Observing a RAG pipeline with OpenTelemetry means tracing both halves of retrieval-augmented generation, the retrieval step (embedding and vector search) and the generation step (the LLM call), over OTLP, so a bad answer can be diagnosed as a retrieval problem or a generation problem, not just labelled slow..
Observing a RAG pipeline with OpenTelemetry means tracing both halves of retrieval-augmented generation, the retrieval step (embedding and vector search) and the generation step (the LLM call), over OTLP, so a bad answer can be diagnosed as a retrieval problem or a generation problem, not just labelled slow.
A retrieval-augmented generation pipeline has two failure surfaces, and confusing them wastes hours. Retrieval fetches context: it embeds the query, searches a vector store, and optionally reranks. Generation passes that context and the query to the LLM to produce an answer. When an answer is wrong, the cause is very often retrieval, the wrong or too few chunks were returned, not the model. So you must observe the retrieval step specifically: what it retrieved, how relevant it was, and how long it took, alongside the usual token, cost and latency of the LLM call.
OpenTelemetry has emerging GenAI semantic conventions for LLM and agent spans, and libraries such as OpenLLMetry and OpenInference instrument the popular frameworks to emit them, with token, model and cost attributes, over OTLP. For LlamaIndex, the instrumentation maps the query engine, the retriever, the embedding call, the vector-store lookup and the synthesis LLM call to spans, so the whole pipeline is one trace. The signals to add on top of the standard gen_ai token and cost attributes are retrieval-specific: the number of chunks retrieved, their scores, and the retrieval latency. This is what lets you separate a slow vector search from a slow model, and a poor answer from an expensive one.
The signals that explain most RAG incidents:
| Signal | Source | What it tells you / watch for |
|---|---|---|
retrieval result count | retriever instrumentation | How many chunks came back; too few (or zero) is the usual cause of a poor, ungrounded answer. |
retrieval span latency | retriever instrumentation | Embedding plus vector-store lookup time; a slow vector store drags the whole request. |
retrieval relevance / scores | retriever instrumentation | The similarity scores of retrieved chunks; low scores mean the context is weak even if plenty was returned. |
gen_ai.usage tokens (context size) | LLM instrumentation | Input tokens reflect how much context was stuffed in; large context drives cost and latency. |
gen_ai.client.operation.duration | LLM instrumentation | Generation latency; grows with context and output size. |
end-to-end pipeline latency | query span | Total query time, split between retrieval and generation so you know which half to fix. |
A RAG dashboard should separate retrieval from generation. Row one is retrieval quality: result count and relevance scores per query type, and retrieval latency, so a weak or slow retriever is obvious. Row two is generation: input tokens (context size), model latency and cost. Row three is the whole pipeline: end-to-end latency split into retrieval versus generation, which tells you where to invest.
# retrieval returning too little (poor answers upstream)
avg(rag.retrieval.result_count) by (query_type)
# retrieval vs generation latency (which half is slow?)
histogram_quantile(0.95, rag.retrieval.duration)
histogram_quantile(0.95, gen_ai.client.operation.duration)
# context size creeping up (cost and latency)
avg(gen_ai.usage.input_tokens) by (query_type)The model is healthy but answers are wrong or vague, because retrieval returned too few chunks or low-relevance ones, so the LLM had nothing good to ground on. Result count and relevance scores expose it. Fix the index, chunking, embedding model or query, not the LLM.
End-to-end latency is high and the retrieval span, not the LLM, dominates, because the vector store is slow (large index, no filtering, cold cache). Retrieval latency isolates it. Tune the vector store, add filtering, or cache.
Cost and latency climb because retrieval is stuffing too much context into the prompt, inflating input tokens. Context size (input tokens) rising is the signal. Cap the number of chunks and rerank so only the most relevant context is passed.
Starting thresholds to tune:
| Condition | Signal | Meaning |
|---|---|---|
| retrieval result count near zero | retrieval | Retrieval returning nothing; answers will be ungrounded. |
| retrieval p95 latency > SLO | retrieval | Vector search is the bottleneck. |
| input tokens (context) > baseline x2 | cost | Context overflow inflating cost and latency. |
| relevance scores dropping | quality | Retrieved context is getting weaker. |
Ops Singularity's AI/ML and Agent Ops pillar operates RAG pipelines in production, ingesting this OpenTelemetry data, watching retrieval quality alongside model cost and latency, and taking governed action when retrieval or generation degrades. Explore AI/ML and Agent Ops and the RAG explainer.
Trace both halves: the retrieval step (embedding, vector search, reranking) and the generation step (the LLM call), capturing retrieval result count and relevance alongside gen_ai token and latency attributes, over OTLP.
Most often it is a retrieval problem, too few or low-relevance chunks, not the model. Observing retrieval result count and relevance scores tells you whether to fix the index and retrieval or the generation.
Capture prompt and completion tokens per call as gen_ai.usage attributes and multiply by the model's price. Because the tokens are on every span, cost rolls up per model, per feature and per request.
Bring a real incident. We will show you Sentinel investigate, act and verify end to end, with every action reversible and audited.