How to Monitor Kubernetes with OpenTelemetry

What to observe: the four layers of a cluster

A complete Kubernetes picture spans four layers, and each answers a different question. The node layer (CPU, memory, disk, PID pressure) tells you whether the hardware can still schedule and run work. The pod and container layer (resource usage against requests and limits, restarts) tells you whether individual workloads are healthy. The cluster/control-plane layer (deployment desired vs available replicas, pod phase, HPA behaviour, scheduler decisions) tells you whether the orchestrator is converging on the state you declared. And the Kubernetes events stream (Pending, Evicted, OOMKilled, CrashLoopBackOff, FailedScheduling) explains why state changed, which the metrics alone never do. Collect all four or you will be guessing.

How OpenTelemetry collects it, and why in two roles

The reason Kubernetes telemetry is collected in two Collector roles is that the signals live at two different scopes. Per-node signals, the CPU, memory, disk and network a kubelet reports for the pods on its node, have to be gathered locally on every node, so the Collector runs as an agent on each node and reads that node's kubelet. Cluster-scoped signals, the desired-versus-available replica counts, pod phases, node conditions and the Kubernetes event stream, exist once for the whole cluster, so a single gateway Collector reads them from the API server. Running the cluster-scoped collection on every node instead would multiply every event and every deployment metric by your node count, which is the most common way these setups end up with wrong numbers.

Three receivers do the work, and it helps to know which signal each one produces. The kubeletstats receiver is the source of resource metrics, per-pod and per-container CPU, memory working set, filesystem and network, read straight from the kubelet. The k8s_cluster receiver is the source of cluster state, it watches the API server and emits the orchestration truth: how many replicas a deployment wants versus has, what phase each pod is in, and whether each node is Ready or under memory, disk or PID pressure. The k8sobjects receiver turns the Kubernetes event stream into logs, so a FailedScheduling, Evicted or OOMKilling event becomes a queryable record rather than something you have to catch live. Binding these together, the k8sattributes processor stamps every metric, log and trace with the pod, namespace, deployment and node it came from, which is what lets you pivot from a slow trace to the exact pod and then to that pod's memory metric without leaving the data.

One operational note that trips people up: the Collector's service account needs cluster read access (list and watch on nodes, pods, events and deployments, plus the kubelet stats subresource). Without that RBAC, the receivers start cleanly but return nothing, so an empty dashboard is more often a permissions problem than a collection one.

Key metrics to watch

These are the OpenTelemetry semantic-convention metric names emitted by the receivers above (exact names can shift slightly between Collector versions, so confirm against your build):

What to put on a dashboard

A useful Kubernetes dashboard answers three questions in order: is the cluster healthy, are workloads getting the resources they asked for, and is anything about to fail. Build it in three rows. The top row is cluster posture, a count of nodes not in the Ready condition, total pods by phase (a rising Pending count is the early tell), and deployments whose available replicas trail their desired count. The middle row is resource pressure, node CPU and memory utilization against allocatable, and, the panel that earns its place, container memory working set as a fraction of each container's limit, because that ratio climbing toward one is the single best predictor of an imminent OOMKill. The bottom row is workload stability, restarts by pod over time and the top pods by restart count, which surfaces crash loops before anyone opens a ticket.

Because the data is OpenTelemetry, the same signals can be expressed as queries against whichever backend holds them, so a memory-pressure panel is conceptually just the working set divided by the limit, grouped by pod:

# memory headroom per pod (values near 1.0 = OOM risk)
container.memory.working_set / k8s.container.memory_limit  by (k8s.pod.name)

# pods stuck Pending longer than is normal for your cluster
count( k8s.pod.phase == "Pending" )  by (k8s.namespace.name)

# crash-looping containers: restarts rising over the last 15m
increase( k8s.container.restarts[15m] )  by (k8s.pod.name)

The exact syntax depends on your query engine, but the shape is the point: every panel is a plain expression over named, correlated signals, not a bespoke agent metric you cannot reproduce elsewhere.

Example alert conditions

Concrete starting thresholds you can tune to your SLOs:

Common failure modes and how to troubleshoot

CrashLoopBackOff

The container starts, exits, and Kubernetes backs off before restarting. Read k8s.container.restarts to confirm the loop, then check the container's last terminated reason from the events stream: OOMKilled points to memory (compare container.memory.working_set to the limit), while Error with a non-zero exit code points to the app, its config, or a failing dependency, found in the pod logs the filelog receiver is already shipping.

OOMKilled

The kernel killed the container for exceeding its memory limit. Confirm by watching container.memory.working_set climb toward the limit before the restart. Fix by right-sizing the limit, fixing the leak, or (for legitimately spiky workloads) tuning requests so the scheduler places the pod where it has headroom.

Pending pods

k8s.pod.phase stays Pending and a FailedScheduling event appears. Compare node allocatable (reported by k8s_cluster) against the sum of pod requests, then check for node taints without matching tolerations, or a PersistentVolumeClaim that never bound.

Node NotReady

k8s.node.condition_ready drops to 0 and MemoryPressure or DiskPressure conditions flip. Check k8s.node.memory.usage and k8s.node.filesystem.usage against capacity; a full disk on a node evicts pods and spreads the incident. Cordon the node, drain it, and reclaim space or replace it.

From monitoring to autonomous resolution

Ops Singularity's TelemetryOps pillar ingests this OpenTelemetry data natively, and Sentinel AI acts on it: a CrashLoopBackOff or node-pressure incident is investigated, resolved through a governed Action Ticket and validated by Sherlock, rather than paging an engineer. Explore TelemetryOps and InfraOps.