How to Instrument Go Services with OpenTelemetry

What to observe in a Go service

For a Go service you watch per-handler latency and error rate, the latency of downstream calls, and the Go runtime: goroutine count, heap allocation, and garbage-collection pauses. Two Go realities shape this: goroutines are cheap to create and easy to leak, so a climbing goroutine count is a leading indicator of a resource leak, and Go services are often high-throughput, so tail latency (p99) matters more than averages.

The most useful correlation is handler latency against goroutine count and GC pauses on one timeline: a latency problem that lines up with a goroutine climb is a leak, and one that lines up with GC is allocation pressure, two very different fixes.

How OpenTelemetry collects it

Unlike Java or Node.js, Go has no bytecode agent, so you wire instrumentation explicitly in code, which is more work but gives precise control. You initialise the OpenTelemetry SDK with an OTLP exporter, add the library instrumentation, otelhttp for HTTP handlers and clients, otelgrpc for gRPC, and the appropriate wrappers for your database driver, and add the runtime instrumentation that emits goroutine, memory and GC metrics.

The discipline that makes or breaks Go tracing is context propagation: you must thread context.Context through your call chain so child spans attach to the request. Where it is passed correctly, a trace shows exactly which downstream call dominates a slow request; where it is dropped, the trace breaks and you lose the breakdown.

Key metrics to watch

The signals that explain most Go service incidents:

What to put on a dashboard

A Go dashboard should let you tell a leak from allocation pressure from a slow dependency. Row one: handler p99 latency and error rate. Row two: goroutine count and heap allocation over time (both should be flat under steady load; a rising trend is a leak), and GC pause time. Row three: downstream call latency by target.

# p99 latency by handler
histogram_quantile(0.99, http.server.request.duration) by (http.route)

# goroutine count: a steady climb under flat load = leak
runtime.go.goroutines

# GC pause time (allocation pressure)
rate(runtime.go.gc.pause_ns[5m])

Reading the signals: common failure modes

Goroutine leak

Memory and file descriptors climb over time and the service eventually degrades, because goroutines are being started but never returning, often blocked on a channel or a request without a timeout. runtime.go.goroutines climbing under steady load is the tell. Add context deadlines and ensure every goroutine has an exit path.

GC pressure

Latency shows periodic spikes that line up with garbage collection, because the service allocates heavily and the GC runs often. runtime.go.gc.pause_ns confirms it. Reduce allocations on the hot path (reuse buffers, avoid unnecessary copies) rather than just adding memory.

Slow dependency

One handler's p99 rises and its downstream span, a query or an API call, accounts for the extra time while the runtime panels stay flat. The fix is in the dependency; the trace makes that unambiguous.

Example alert conditions

Starting thresholds to tune:

From monitoring to autonomous resolution

Ops Singularity's TelemetryOps ingests Go OpenTelemetry data natively, and Sentinel AI resolves the service incidents it surfaces through governed Action Tickets. Explore TelemetryOps.