How to Observe Terraform-Managed Infrastructure

What to observe, and why Terraform is a special case

Terraform is a provisioning tool, not a runtime component, so there is no Terraform receiver and no live metric stream from Terraform itself. The honest framing is that you observe two other things: the delivery pipeline that runs plan and apply, and the infrastructure Terraform provisions, which you monitor with the ordinary OpenTelemetry receivers for that cloud or platform.

So the operational questions are: did the last apply succeed and how long did it take, has the real infrastructure drifted from the declared state, and are the provisioned resources healthy and within budget. Those are answered by pipeline telemetry and by the runtime telemetry of the resources, not by Terraform.

How OpenTelemetry collects it

For the pipeline, instrument the CI job that runs Terraform so each plan and apply emits a span with its outcome and duration over OTLP, giving you a trace of every infrastructure change, who ran it, against which workspace, and whether it succeeded. For the resulting infrastructure, collect telemetry through the appropriate receivers, cloud metrics receivers for managed services, the Kubernetes receivers for clusters Terraform created, host metrics for VMs it provisioned.

Drift detection is its own signal: run terraform plan on a schedule and emit whether it would change anything, so a non-empty plan against a supposedly stable environment flags that reality has diverged from code. The value of tracing applies is correlation, when a resource's behaviour or cost changes, you can line it up with the exact apply that changed it.

Key metrics to watch

The signals worth tracking around Terraform-managed infrastructure:

What to put on a dashboard

A Terraform-adjacent dashboard has two halves. The change half: apply success and failure over time, apply duration, and a drift indicator from scheduled plans. The resource half: the health and cost of the provisioned infrastructure, from its own telemetry, with a way to overlay when applies happened so a change in behaviour or cost lines up with the apply that caused it.

# apply outcomes over time (from pipeline spans)
count(terraform.apply.result)  by (workspace, result)

# drift: scheduled plans that would change something
count(terraform.plan.changes > 0)  by (workspace)

# correlate a resource metric with the apply that changed it
# overlay apply span timestamps on the resource metric timeline

Reading the signals: common failure modes

Failed or partial apply

An apply fails or applies partially, leaving infrastructure in a state that does not match the code and may be internally inconsistent. The apply span status flags it. Because a partial apply can leave real resources half-changed, treat a failed apply as an incident, not a retry-and-forget, and reconcile state before the next change.

Configuration drift

The real infrastructure diverges from what the code declares, because someone changed a resource in the console, so the next apply will either revert their change or fail. A scheduled plan that is not empty is the detector. Investigate the out-of-band change and bring it back under Terraform rather than letting drift accumulate.

A change that breaks or inflates cost

A resource starts misbehaving or its cost jumps, and the cause is a specific Terraform apply, a resized instance, a changed configuration. Overlaying apply timestamps on the resource's own telemetry ties the effect to the change, which is the whole point of tracing applies: infrastructure incidents get a change to blame.

Example alert conditions

Starting conditions to tune:

From monitoring to autonomous resolution

Ops Singularity observes the infrastructure Terraform provisions through TelemetryOps and resolves incidents on those resources through governed Action Tickets, while linking them back to the change that caused them. Explore TelemetryOps and InfraOps.