How to Monitor Docker Containers with OpenTelemetry

What to observe in Docker containers

For containers running directly on Docker rather than under Kubernetes, the signals that matter are per-container CPU against any CPU quota, memory usage against the container limit, network and block I/O, restart counts, and health-check status. Because containers are ephemeral and are restarted silently, the value is watching these against the container lifecycle, a container that keeps restarting or keeps hitting its memory limit is telling you something the aggregate host metrics hide.

The single most useful number is memory usage as a percentage of the container's limit, because a container that reaches its limit is killed by the kernel without a graceful shutdown, and that is the most common cause of a container that 'just disappears'.

How OpenTelemetry collects it

The OpenTelemetry Collector's docker_stats receiver connects to the Docker daemon and reads the same per-container statistics the docker stats view shows, CPU, memory against the limit, network and block I/O, exporting them over OTLP. Container logs are collected with the filelog receiver, and applications inside containers export their own traces and metrics over OTLP as normal.

The access detail to get right is the daemon socket: the Collector needs read access to the Docker socket to call the stats API, so mount it read-only and scope the permission carefully rather than running the Collector with broad privileges.

Key metrics to watch

The signals that explain most container incidents:

What to put on a dashboard

A Docker dashboard should make a memory or restart problem obvious at the container level. Row one: memory percent of limit per container (the OOM predictor) and CPU utilization per container. Row two: restart counts over time and the top containers by restarts. Row three: network and block I/O per container, so a noisy neighbour starving the host is visible.

# memory headroom per container (near 1.0 = OOM kill risk)
container.memory.percent  by (container.name)

# CPU saturation against the container quota
container.cpu.utilization  by (container.name)

# block I/O per container (spot the noisy neighbour)
rate(container.blockio.io_service_bytes_recursive[5m])  by (container.name)

Reading the signals: common failure modes

OOM kill

A container vanishes and restarts with no graceful shutdown, because it exceeded its memory limit and the kernel killed it. container.memory.percent climbing to 100 just before the restart is the tell. Raise the limit if the workload legitimately needs it, or fix the leak driving usage up.

CPU throttling

A container's latency rises while its CPU sits pinned at its quota. Utilization against the quota is saturated even though the host has spare capacity, because the container's CPU limit is throttling it. Raise the limit or right-size the workload.

Restart loop

A container restarts repeatedly, so restart count climbs steadily. The cause is either the application crashing on startup (check the logs the filelog receiver is shipping) or the container being OOM-killed each time it reaches its limit.

Example alert conditions

Starting thresholds to tune:

From monitoring to autonomous resolution

Ops Singularity's TelemetryOps ingests Docker telemetry over OpenTelemetry, and Sentinel AI resolves container incidents, restarting, rescheduling or applying a fix, through governed Action Tickets. Explore TelemetryOps.