How to Observe CrewAI Multi-Agent Workflows with OpenTelemetry

What to capture in a multi-agent workflow

A CrewAI workflow is a crew of agents collaborating on tasks, and its behaviour is only intelligible if you can see the structure: a span for the crew run, a span per agent and per task, and, inside each, the LLM calls, tool calls and handoffs. The signals that matter are tokens and cost per agent (multi-agent workflows can be expensive because each agent calls a model), task duration and success, and, critically, the number of steps an agent takes, because a common failure mode is an agent looping and re-calling the model without converging.

How to instrument CrewAI

OpenTelemetry has emerging GenAI semantic conventions for LLM and agent spans, and libraries such as OpenLLMetry and OpenInference instrument the popular frameworks to emit them, with token, model and cost attributes, over OTLP. Mapping CrewAI's Crew, Agent and Task objects to spans gives you the hierarchy, and the LLM instrumentation underneath captures the token and cost attributes per call. One CrewAI-specific detail: the framework emits its own anonymous usage telemetry, which you should tell apart from your application tracing, instrument your own spans deliberately so your observability reflects your workflow, not the framework's internal metrics.

Key metrics to watch

The signals that explain most multi-agent incidents:

What to put on a dashboard

A CrewAI dashboard should surface cost and loops first. Row one: total cost per crew run and tokens per agent, so an expensive agent is obvious. Row two: steps per agent over time (the runaway-loop detector) and task duration and success rate. Row three: tool call latency and errors, since a failing tool is a frequent cause of agents spinning.

# cost per crew run (multi-agent runs add up fast)
sum(gen_ai.usage.input_tokens * price_in + gen_ai.usage.output_tokens * price_out) by (crew.run_id)

# steps per agent: an outlier is a runaway loop
max(agent.step.count) by (agent.name, crew.run_id)

# tool failures driving agent retries
rate(tool.call{result="error"}[5m]) by (tool.name)

Reading the signals: common failure modes

Runaway agent loop

Cost and latency for a crew run spike because an agent keeps reasoning and re-calling the model without reaching a conclusion. The step count for that agent is far above normal. Bound agent iterations, improve the task definition, or add a stopping condition; the step-count metric catches it before the bill does.

Cost blow-up across agents

A workflow that seemed cheap in testing is expensive in production because every agent calls a model and the tokens multiply. Per-agent token attribution shows which agent dominates cost, so you can trim its prompts, reduce its context, or consolidate agents.

A stuck or failing task

One task stalls, and because agents depend on each other, the whole crew stops making progress. Task duration and status pinpoint the blocked task, and the tool and LLM spans inside it show whether a failing tool or a model error is the cause.

Example alert conditions

Starting thresholds to tune:

From monitoring to autonomous resolution

Ops Singularity's AI/ML and Agent Ops pillar operates multi-agent systems in production, ingesting this OpenTelemetry data, watching per-agent cost and step counts, and taking governed action, bounding a runaway agent, escalating a stuck crew, when a workflow degrades. Explore AI/ML and Agent Ops.