How to Monitor CI/CD Pipelines with OpenTelemetry

Monitoring CI/CD pipelines with OpenTelemetry means tracing pipeline runs and their stages as spans over OTLP, so build and delivery health, and its bottlenecks, become observable.

What to observe in a CI/CD pipeline

A slow or flaky CI/CD pipeline taxes every engineer, but pipelines are often a blind spot. Treating a pipeline run as a distributed trace, with each stage, job and step as a span, exposes where time actually goes, which steps fail, how flaky the pipeline is, and how long work waits for a runner. The signals that matter are pipeline duration and its trend, stage and job failure rates, flaky-test rate, and queue or runner wait time.

The most useful reading is duration broken down by stage: a pipeline that has slowly crept from five minutes to fifteen almost always has one or two stages responsible, and the span breakdown names them.

How OpenTelemetry collects it

OpenTelemetry has a growing CI/CD observability effort, and many CI systems can emit trace data or be instrumented to. You instrument the pipeline so each run produces a trace with spans for its stages, jobs and steps, including durations and outcomes, exported over OTLP, and collect build and test logs with the filelog receiver for detail.

Because CI/CD tracing conventions are still maturing, the practical approach is to emit a span per stage and step with consistent attributes (pipeline, branch, result) so you can aggregate across runs; that consistency is what turns individual traces into trend data.

CI/CD span and metric conventions are still evolving in OpenTelemetry; the signals below are derived from per-run spans.

Key metrics to watch

The signals that explain most pipeline problems, derived from per-run traces:

Signal	How to capture	What it tells you / watch for
`pipeline duration`	root span duration	Total time per run; the trend matters more than any single run.
`stage / step duration`	child spans	Where time goes; a slowly growing stage is the usual cause of a slowing pipeline.
`stage / job failure rate`	span status	Which steps fail and how often; separates real breakage from flakiness.
`flaky-test rate`	test result spans/logs	Intermittent failures that pass on retry; a productivity and trust drain.
`queue / runner wait time`	span between trigger and start	Time a run waits for a runner; high wait means runner capacity, not pipeline logic, is the bottleneck.
`success rate by branch`	span attributes	Whether main is healthier than feature branches, and where breakage concentrates.

What to put on a dashboard

A pipeline dashboard should make slow and flaky obvious. Row one: pipeline duration trend and a breakdown of duration by stage. Row two: failure rate by stage and flaky-test rate. Row three: runner queue wait time and success rate by branch. Together these tell you whether to fix a slow stage, a flaky test, or add runner capacity.

# pipeline duration trend (are runs getting slower?)
avg(pipeline.run.duration)  by (pipeline)

# slowest stages (where the time actually goes)
avg(pipeline.stage.duration)  by (stage)

# runner wait time (capacity bottleneck, not pipeline logic)
avg(pipeline.queue.wait_time)  by (pipeline)

Reading the signals: common failure modes

Slowly growing pipeline

Total duration creeps up over weeks until the pipeline is a bottleneck. The stage-duration breakdown shows which one or two stages grew, dependency install, a test suite, an image build, so you fix the actual culprit rather than optimising blindly. Caching and parallelisation usually target the identified stage.

Flaky tests

Runs fail intermittently and pass on retry, eroding trust in the pipeline until people ignore failures. Tracking the flaky-test rate across runs surfaces the specific tests. Quarantine and fix them; a pipeline whose failures are ignored is worse than no pipeline.

Runner starvation

Runs sit waiting before they even start, so total lead time is high even though the pipeline itself is fast. Queue-wait time is the signal, and it points at runner capacity rather than pipeline logic. Add runners or tune concurrency.

Example alert conditions

Starting thresholds to tune:

Condition	Signal	Meaning
pipeline duration trend up over 2 weeks	speed	The pipeline is slowing; find the stage.
stage failure rate rising	reliability	A step is breaking repeatedly.
flaky-test rate > threshold	trust	Intermittent failures eroding confidence.
runner wait time high	capacity	Add runners; capacity is the bottleneck.

From monitoring to autonomous resolution

Ops Singularity's DevSecOps and TelemetryOps capabilities observe delivery pipelines and can act on failures and risk through governed change gates and Action Tickets. Explore DevSecOps and TelemetryOps.

Frequently asked questions

Can you trace a CI/CD pipeline with OpenTelemetry?

Yes. Instrument the pipeline so each run is a trace with spans for stages, jobs and steps, including durations and outcomes, exported over OTLP, with logs collected via the filelog receiver.

What CI/CD signals matter most?

Pipeline duration and its trend, stage and job failure rates, flaky tests, and queue or runner wait time.