How to Monitor GPU and AI Training Workloads

What to observe on GPUs and AI training jobs

GPUs are expensive and usually the bottleneck, so utilization is the headline: GPU and memory utilization, temperature and power draw, memory used against capacity, and hardware errors. For training runs, throughput (samples or tokens per second) and step time tell you whether the GPUs are actually saturated or starved. The two costly failure modes are under-utilized GPUs (wasted spend) and out-of-memory failures that kill a run partway.

The reading that matters most for cost is GPU utilization against memory utilization: low compute utilization while memory is full often means the job is input-bound (data loading, not the GPU), and low on both means the expensive hardware is simply idle.

How OpenTelemetry collects it

NVIDIA's DCGM exporter exposes GPU metrics in Prometheus format; you scrape it with the Collector's prometheus receiver and export over OTLP, so GPU telemetry joins the same pipeline as everything else. The training job itself is instrumented to emit spans and metrics per run and epoch, so hardware-level and run-level telemetry are correlated.

The deployment detail is running the DCGM exporter on every GPU node (a DaemonSet under Kubernetes, typically via the GPU operator) so every accelerator is covered, and labelling metrics with the job and run so utilization ties back to a specific workload.

Key metrics to watch

The signals that explain most GPU and training incidents:

What to put on a dashboard

A GPU dashboard should make waste and risk obvious. Row one: GPU utilization and memory utilization per device side by side (low compute with full memory signals an input-bound job). Row two: memory used against capacity (the OOM predictor), temperature and power. Row three: training throughput and step time, and XID error counts.

# GPU compute utilization (low on costly hardware = waste)
DCGM_FI_DEV_GPU_UTIL  by (gpu, node)

# memory headroom (used approaching capacity = OOM risk)
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)

# hardware errors that crash runs
increase(DCGM_FI_DEV_XID_ERRORS[1h])  by (gpu, node)

Reading the signals: common failure modes

Under-utilized GPUs

GPU compute utilization sits low while the job runs, meaning expensive hardware is idle, often because the training pipeline is input-bound (data loading or preprocessing cannot feed the GPU) or batch size is too small. Compare compute utilization to memory utilization; the fix is in the data pipeline or batching, not more GPUs.

Out-of-memory failure

A run dies partway because framebuffer memory reached capacity, usually from too large a batch size or model for the device. Used memory climbing to capacity just before the failure is the tell. Reduce batch size, use gradient checkpointing or sharding, or move to a larger-memory device.

Thermal throttling or XID errors

Throughput drops because a GPU is throttling on temperature, or a run crashes with XID errors indicating a hardware or driver fault. Temperature trending high, or XID error counts rising, identifies the failing device; cordon it and check cooling or the driver.

Example alert conditions

Starting thresholds to tune:

From monitoring to autonomous resolution

Ops Singularity's TelemetryOps ingests GPU telemetry over OpenTelemetry and its FinOps pillar ties utilization to cost, while Sentinel AI acts on inefficient or failing runs through governed Action Tickets. Explore TelemetryOps and AI/ML and Agent Ops.