How to Monitor Apache Kafka with OpenTelemetry

What to observe in Apache Kafka

The defining Kafka signal is consumer-group lag, how far each consumer group is behind the newest offset on each topic and partition, because lag that grows without bound means data is not being processed in time, and everything downstream goes stale. Around it you watch the broker fabric: under-replicated partitions, offline partitions, request latency, throughput, and broker disk usage.

Reading lag correctly means reading it per partition, not just as a group total: a single hot or stuck partition can hide inside a healthy-looking aggregate, and it is usually the partition, not the whole group, that is the incident.

How OpenTelemetry collects it

The OpenTelemetry Collector's kafkametrics receiver connects to the brokers and collects the fabric-level and consumer-level metrics, broker and partition counts, per-partition current and oldest offsets, in-sync replica counts, and consumer-group lag and offsets, without running an agent on the brokers. For broker JVM and internal metrics, the JMX receiver reads Kafka's MBeans over a JMX port.

Producers and consumers instrumented with OpenTelemetry add the other half: traces that follow a message from producer through the broker to the consumer, so you can see not just that lag exists but where in the pipeline a message stalls. The setup detail to get right is pointing kafkametrics at the bootstrap brokers with the correct protocol and credentials; consumer lag is derived from the difference between the partition's latest offset and the group's committed offset, so both must be readable.

Key metrics to watch

The signals that explain most Kafka incidents:

What to put on a dashboard

A Kafka dashboard should surface lag and replication health first. Row one: consumer-group lag over time per group and topic (a rising line is the alarm), and the top partitions by lag to catch the hot-partition case. Row two: under-replicated partition count derived from replicas minus in-sync replicas, and offline partitions. Row three: throughput per topic and broker disk usage, so a stalled pipeline or a filling disk is visible early.

# consumer lag by group and topic (watch the slope, not the value)
kafka.consumer_group.lag by (group, topic)

# under-replicated partitions = replicas that are not in sync
kafka.partition.replicas - kafka.partition.replicas_in_sync

# throughput per partition (a drop can mean a stalled consumer)
rate(kafka.partition.current_offset[5m]) by (topic, partition)

Reading the signals: common failure modes

Growing consumer lag

Lag climbs steadily for a group while producers keep writing, so consumers are falling behind. Read it per partition: if one partition dominates, the consumer handling it is slow or stuck; if all rise together, the group is under-provisioned or repeatedly rebalancing. Scale consumers, fix the slow processing, or stop the rebalance loop.

Under-replicated partitions

The count of partitions whose replicas are not all in sync rises, meaning a broker is down or too slow to keep up, and durability is at risk. Identify the lagging broker from the fabric metrics; the cause is usually a broker under disk or network pressure, or a broker that has fallen out of the cluster.

Rebalance storms

Consumer-group membership churns and lag sawtooths as the group repeatedly rebalances, pausing consumption each time. The tell is members joining and leaving in the metrics. Causes include consumers exceeding the session timeout because processing is too slow, or unstable instances; tune timeouts and processing, or stabilise the consumers.

Example alert conditions

Starting thresholds to tune:

From monitoring to autonomous resolution

Ops Singularity's TelemetryOps ingests Kafka telemetry over OpenTelemetry, and its DataOps and Sentinel capabilities resolve streaming incidents, growing lag, a stalled consumer, an under-replicated partition, through governed Action Tickets. Explore TelemetryOps and DataOps.