How to Instrument Python Apps with OpenTelemetry

What to observe in a Python service

For a Python web service the signals that matter are per-endpoint latency and error rate, the latency of database queries and external calls the request makes, and unhandled exceptions. Two Python-specific realities shape what you watch: many frameworks run under multiple worker processes (gunicorn or uvicorn), so telemetry has to be initialised per worker, and CPU-bound work holds the GIL, so a single hot request can starve the whole worker.

The most valuable thing distributed traces give a Python service is the breakdown inside a slow request, whether the time is spent in your code, in a database query, or waiting on a downstream API, which is exactly what turns a vague 'the endpoint is slow' into a fix.

How OpenTelemetry collects it

Python supports zero-code instrumentation through the opentelemetry-instrument agent. You install opentelemetry-distro, run opentelemetry-bootstrap to pull in the instrumentations for the libraries you actually use, Django, Flask, FastAPI, requests, psycopg, SQLAlchemy, and launch the app under the agent with an OTLP exporter. Every request becomes a server span and every instrumented call a child span, so the request breakdown comes for free.

The gotcha that silently breaks Python setups is worker forking: gunicorn and uvicorn fork worker processes, and the tracer must be initialised in each worker, not only the parent, or you get no spans from the workers. The fix is to initialise in a post-fork hook. For your own logic, the SDK adds manual spans and custom metrics; for CPython runtime health, the process-runtime instrumentation emits CPU, memory and garbage-collection metrics.

Key metrics to watch

The signals that explain most Python service incidents:

What to put on a dashboard

A Python dashboard should separate 'my code is slow' from 'my database is slow' from 'my workers are saturated'. Row one: endpoint p95/p99 and 5xx rate by route. Row two: database time as a share of request time, and the number of DB spans per request to catch N+1 patterns. Row three: per-worker CPU and memory, so GIL-bound saturation and leaks are visible before requests start timing out.

# p95 latency by endpoint
histogram_quantile(0.95, http.server.request.duration) by (http.route)

# database time as a fraction of total request time
sum(db.client.operation.duration) by (http.route) / sum(http.server.request.duration) by (http.route)

# worker CPU saturation (GIL-bound work)
process.runtime.cpython.cpu.utilization by (worker)

Reading the signals: common failure modes

N+1 queries

One endpoint is slow and its trace contains dozens of near-identical database spans, one per row of a collection. Database time dominates the request. The fix is eager loading or a single batched query, and the trace makes the pattern obvious in a way logs never would.

Blocking call in an async loop

A FastAPI or asyncio service degrades under load because a synchronous call, a blocking DB driver or a CPU-bound function, is running on the event loop. Latency rises across all routes on that worker while CPU spikes. Move the call to a threadpool or use an async driver.

Worker saturation

All workers are busy and requests queue, so latency climbs with no single slow dependency. Per-worker CPU is pinned. Add workers, or move CPU-bound work out of the request path.

Example alert conditions

Starting thresholds to tune:

From monitoring to autonomous resolution

Ops Singularity's TelemetryOps ingests Python OpenTelemetry data natively, and Sentinel AI resolves the service incidents it surfaces through governed Action Tickets. Explore TelemetryOps.