Monitoring Envoy with OpenTelemetry means collecting the metrics, access logs and traces Envoy natively supports and exporting them over OTLP, so your proxy and mesh data plane is fully visible..
Monitoring Envoy with OpenTelemetry means collecting the metrics, access logs and traces Envoy natively supports and exporting them over OTLP, so your proxy and mesh data plane is fully visible.
Envoy is a programmable proxy at the heart of many meshes and gateways, and it is observability-rich by design: per-cluster (upstream) request rate, latency and response codes, retries and timeouts, circuit-breaker events, connection-pool usage, and upstream health-check status. These signals show exactly how traffic flows and, when something fails, which upstream Envoy is protecting you from.
The most actionable reading is per-upstream-cluster error rate and latency together with retry and circuit-breaker state, because Envoy often masks a failing upstream with retries until the breaker opens, and seeing that sequence early is the difference between a contained incident and a cascade.
Envoy has native OpenTelemetry support: it can emit distributed traces through an OpenTelemetry tracer and export access logs through the OpenTelemetry access-log service, both to a Collector over OTLP. Its extensive statistics are scraped with the Collector's prometheus receiver, which is where the per-cluster request, retry and circuit-breaker metrics come from.
The setup is configuring an OpenTelemetry tracing provider and the access-log service in the Envoy config, and pointing the prometheus receiver at Envoy's admin stats endpoint. Applications must propagate trace-context headers for the traces to span multiple hops.
The signals that explain most Envoy incidents:
| Signal | Source | What it tells you / watch for |
|---|---|---|
envoy_cluster_upstream_rq (by response code) | Envoy stats | Requests per upstream by status; the 5xx share per cluster is your error signal. |
envoy_cluster_upstream_rq_time | Envoy stats | Upstream latency histogram per cluster; the basis for p95/p99 per dependency. |
envoy_cluster_upstream_rq_retry | Envoy stats | Retries to an upstream; a surge means that upstream is failing. |
envoy_cluster_upstream_rq_timeout | Envoy stats | Timeouts to an upstream; a slow or dead dependency. |
circuit breaker open (cx/rq) | Envoy stats | An open breaker means Envoy has stopped sending traffic to an overloaded upstream. |
envoy_cluster_upstream_cx_active | Envoy stats | Active connections to an upstream; near the pool limit means connection-pool exhaustion. |
An Envoy dashboard is organised per upstream cluster. Row one: request rate and 5xx share per cluster, and p95/p99 upstream latency per cluster. Row two: retries and timeouts per cluster, and circuit-breaker open state, the sequence that reveals a failing upstream. Row three: connection-pool active-versus-limit and upstream health-check status.
# 5xx share per upstream cluster
sum(rate(envoy_cluster_upstream_rq{response_code_class="5"}[5m])) by (envoy_cluster_name)
/ sum(rate(envoy_cluster_upstream_rq[5m])) by (envoy_cluster_name)
# retries surging = an upstream is failing
rate(envoy_cluster_upstream_rq_retry[5m]) by (envoy_cluster_name)
# connection-pool pressure per upstream
envoy_cluster_upstream_cx_active by (envoy_cluster_name)5xx to a cluster rises and retries surge as Envoy retries the failing upstream, which can amplify load. The retry metric moves before the user-visible error rate. Fix the failing upstream and review the retry budget so retries do not overwhelm it.
Traffic to an upstream drops and its breaker is open, because Envoy detected overload or failures and stopped sending requests to protect the system. The breaker is doing its job; the incident is the unhealthy upstream that tripped it.
Requests to an upstream queue or fail while active connections sit at the pool limit, because the pool is exhausted. The active-connections metric against the configured limit is the tell. Raise the pool size or address why upstream responses are slow enough to hold connections open.
Starting thresholds to tune:
| Condition | Signal | Meaning |
|---|---|---|
| upstream 5xx share > 2% for 5m | errors | An upstream cluster is failing. |
| retries to a cluster rising for 5m | dependency | An upstream is failing and being retried. |
| circuit breaker open | availability | Envoy has isolated an overloaded upstream. |
| upstream cx active at pool limit | saturation | Connection-pool exhaustion. |
Ops Singularity's TelemetryOps ingests Envoy telemetry over OpenTelemetry, and Sentinel AI resolves the upstream incidents it exposes through governed Action Tickets. Explore TelemetryOps.
Yes. Envoy can emit distributed traces through an OpenTelemetry tracer and export access logs via the OpenTelemetry access-log service to a Collector over OTLP, and its metrics can be scraped with the prometheus receiver.
Per-cluster request rate, latency and response codes, retries and timeouts, circuit-breaker trips, and connection-pool and upstream health.
Bring a real incident. We will show you Sentinel investigate, act and verify end to end, with every action reversible and audited.