How to Monitor Application Performance with OpenTelemetry

What APM signals to capture

Application performance monitoring comes down to a small set of request-centric signals plus the traces behind them. Per endpoint, you want latency (as p95/p99, not averages), error rate and throughput, the RED method. Underneath, distributed traces show where a request spent its time across services and downstream calls, and runtime metrics (garbage collection, event-loop lag, thread pools, depending on the language) explain latency the application code alone does not. Together these answer not just whether an app is slow, but which request and which call is responsible.

How OpenTelemetry replaces the APM agent

Where classic APM used a proprietary per-language agent, OpenTelemetry gives you open auto-instrumentation for most languages that produces the same traces and metrics with little or no code change. The OpenTelemetry Collector is the workhorse here: it receives telemetry over OTLP and exports it to your backend, so instrumentation stays vendor-neutral and portable. You instrument the service, the Collector receives the OTLP data, and the same telemetry can go to any backend, so you own the data and avoid lock-in. The span is the unit of APM: every request becomes a trace, and every downstream call a child span you can measure.

Key metrics to watch

The signals that explain almost every application-performance incident:

What to put on a dashboard

An APM dashboard should let you decide in seconds whether a latency problem is in your code, a dependency, or the runtime. Row one is request health: p95/p99 latency by endpoint and error rate by endpoint. Row two is dependencies: latency of downstream calls and database queries, grouped by target, so a slow dependency is obvious. Row three is the runtime and the trace view: GC or event-loop health, and a service map derived from traces so you can follow a slow request across services.

# p99 latency by endpoint (find the slow route, not "the app")
histogram_quantile(0.99, http.server.request.duration) by (http.route)

# error rate by endpoint
sum(rate(http_server_requests{status=~"5.."}[5m])) by (http.route)
  / sum(rate(http_server_requests[5m])) by (http.route)

# database time as a share of request time (spot DB-bound endpoints)
sum(db.client.operation.duration) by (http.route) / sum(http.server.request.duration) by (http.route)

Reading the signals: common failure modes

Slow endpoint from a downstream call

One endpoint's p99 rises and its trace shows a specific downstream span, a query or an API call, accounting for the added time while the app code itself is fast. The fix is in the dependency; the trace makes that unambiguous rather than leaving you to guess.

N+1 queries

An endpoint is slow and its trace contains many near-identical database spans, one per item in a collection. Database time dominates. The fix is eager loading or a single batched query, and only the trace makes the pattern obvious.

Runtime, not code

Latency spikes with no matching change in request logic, because the runtime is the cause, GC pauses in a JVM or .NET service, event-loop lag in Node.js, thread-pool starvation. Overlaying runtime metrics on the latency histogram distinguishes this from a slow dependency.

Example alert conditions

Reasonable starting thresholds to tune to your SLOs:

From monitoring to autonomous resolution

Ops Singularity's TelemetryOps pillar ingests this OpenTelemetry APM data natively, and Sentinel AI resolves the incidents it reveals, a slow endpoint, an N+1 query, a failing dependency, a runtime problem, through governed Action Tickets validated by Sherlock. Explore TelemetryOps and ServiceOps.