Observing Istio with OpenTelemetry means collecting the traces, metrics and access logs the mesh's Envoy sidecars produce and exporting them over OTLP, so service-to-service traffic across the mesh is fully visible..
Observing Istio with OpenTelemetry means collecting the traces, metrics and access logs the mesh's Envoy sidecars produce and exporting them over OTLP, so service-to-service traffic across the mesh is fully visible.
A service mesh sees every request between services, which makes it the richest source of the golden signals: request rate, error rate and latency percentiles per service and per route, plus mesh-specific signals, retries, timeouts, circuit-breaker trips, and mTLS handshake success. The unique value is the call graph: when latency or errors appear, the mesh shows you which hop between which two services is actually responsible.
The reading that saves the most time is per-hop latency: a request that is slow end to end almost always has one edge in the call graph where the time is spent, and the mesh telemetry isolates it without touching application code.
Istio's data plane is Envoy sidecars, and they are the telemetry source. Configure Istio's telemetry to emit distributed traces and export them, along with Envoy access logs, to an OpenTelemetry Collector over OTLP, and scrape the Istio and Envoy metrics with the Collector's prometheus receiver. Application spans propagate through the mesh as long as services forward the standard trace-context headers, which is the one thing the mesh cannot do for you.
Two setup details matter: enable tracing in the mesh config and set a sampling rate you can afford (100% tracing on a busy mesh is expensive), and confirm your applications propagate the context headers, or traces will break at every hop and you lose the call graph that makes the mesh worth instrumenting.
The signals that explain most mesh incidents:
| Signal | Source | What it tells you / watch for |
|---|---|---|
istio_requests_total | Istio telemetry | Request rate and, by response_code label, error rate per source and destination service. |
istio_request_duration_milliseconds | Istio telemetry | Latency histogram per service and route; the basis for p95/p99 per hop. |
envoy_cluster_upstream_rq_retry | Envoy stats | Retries to an upstream; a surge means an upstream is failing and the mesh is masking it. |
envoy_cluster_upstream_rq_timeout | Envoy stats | Timeouts to an upstream; a slow or dead dependency. |
circuit breaker open state | Envoy stats | An open breaker means the mesh has stopped sending traffic to an unhealthy upstream. |
mTLS handshake failures | Istio / Envoy | Failed mutual-TLS handshakes, a config or certificate problem breaking service-to-service calls. |
A mesh dashboard is organised around the golden signals per service, then the call graph. Row one: request rate, error rate and p95/p99 latency per service, from istio_requests_total and the duration histogram. Row two: retries and timeouts per upstream, and circuit-breaker state, the signals that reveal a failing dependency the mesh is protecting you from. Row three: the service dependency graph with per-edge latency, so you can find the slow hop directly.
# error rate per destination service (5xx share of requests)
sum(rate(istio_requests_total{response_code=~"5.."}[5m])) by (destination_service)
/ sum(rate(istio_requests_total[5m])) by (destination_service)
# p99 latency per route (find the slow hop)
histogram_quantile(0.99, istio_request_duration_milliseconds) by (destination_service)
# retries surging to an upstream = a failing dependency
rate(envoy_cluster_upstream_rq_retry[5m]) by (cluster_name)End-to-end latency rises, and the per-hop breakdown shows a single service-to-service edge accounting for the added time. The mesh call graph points straight at the slow upstream; investigate that service rather than the caller reporting the symptom.
A surge in retries to an upstream means that upstream is returning errors and the mesh is retrying, which can amplify load and make the incident worse. The retry metric spikes before the user-visible error rate does. Fix the failing upstream and review the retry policy so retries do not overwhelm it.
Traffic to an upstream drops to zero and its breaker is open, because the mesh detected the upstream as unhealthy and stopped sending requests to protect the system. This is the mesh working as designed; the real incident is the unhealthy upstream that tripped it.
Calls between two services start failing with handshake errors after a certificate rotation or policy change. mTLS failure counts rise on a specific pair. The cause is a certificate or PeerAuthentication misconfiguration, not the application.
Starting thresholds to tune:
| Condition | Signal | Meaning |
|---|---|---|
| destination 5xx rate > 2% for 5m | errors | A service is failing within the mesh. |
| p99 per hop > SLO for 10m | latency | A specific hop is slow. |
| retries to an upstream rising for 5m | dependency | An upstream is failing and being retried. |
| circuit breaker open | availability | The mesh has isolated an unhealthy upstream. |
Ops Singularity's TelemetryOps ingests Istio and Envoy telemetry over OpenTelemetry, and Sentinel AI correlates a mesh-level incident to its cause and resolves it through a governed Action Ticket. Explore TelemetryOps.
Yes. Istio's Envoy sidecars can emit distributed traces and export telemetry through OpenTelemetry, so mesh traffic can be sent to an OpenTelemetry Collector over OTLP alongside application spans.
Per-service request rate, latency and error rate, retries and circuit-breaker trips, mTLS status, and the service call graph, so you can see where latency or errors originate.
Bring a real incident. We will show you Sentinel investigate, act and verify end to end, with every action reversible and audited.