Monitoring cron and batch jobs means instrumenting each run to emit a span and outcome over OpenTelemetry, so scheduled work that usually runs silently becomes observable and its failures are caught..
Monitoring cron and batch jobs means instrumenting each run to emit a span and outcome over OpenTelemetry, so scheduled work that usually runs silently becomes observable and its failures are caught.
Scheduled jobs, cron tasks, nightly batches, ETL runs, usually run unattended and fail silently, which is exactly why they are a blind spot: a job that did not start, ran too long, or exited with an error can go unnoticed until something downstream breaks. There is no receiver that watches a job for you; the job has to report on itself, so the approach is instrumentation, not scraping.
The two questions to answer for every scheduled job are simply: did it run when it should have, and did it succeed on time. Everything else, duration, records processed, downstream freshness, hangs off those two.
You wrap each job execution in an OpenTelemetry span that records its start and end time, exit status and key attributes (records processed, bytes written), and export it over OTLP, which turns a silent schedule into a stream of run records you can query, alert, and act on. For jobs on Kubernetes, the k8sobjects receiver additionally captures Job and CronJob events, so a job that failed to schedule is visible even if it never ran.
The one thing self-reporting cannot catch is a job that never started, because a job that does not run emits no span. So you also track expected schedules and alert on the absence of a run, comparing runs seen against runs expected, which is the only way to detect a missed job.
The signals that catch most batch-job failures:
| Signal | How to capture | What it tells you / watch for |
|---|---|---|
run occurred | per-run span / scheduler event | Whether the job ran when scheduled; the absence of a span is itself the signal. |
exit status | span status | Success or failure of each run; a non-zero exit is a failed job. |
run duration | span duration | How long the job took; a run exceeding its normal window is an overrun. |
records / bytes processed | span attributes | Throughput; a run that processes far fewer records than usual may have partially failed. |
schedule adherence | expected vs observed runs | A run that should have happened but did not, the failure self-reporting cannot catch. |
downstream data freshness | dependent dataset timestamp | Whether the data the job produces is up to date, the business impact of a missed run. |
A batch-job dashboard is built around presence and outcome. Row one: a grid of jobs showing last-run time and status, with missed runs highlighted (expected but not seen). Row two: run duration against each job's normal window, to catch overruns, and records processed per run. Row three: downstream data freshness, so the business impact of a late or failed job is visible.
# jobs that ran vs jobs that should have run (detect the missing)
count(job.run{result="success"}) by (job) # compare to expected schedule
# overruns: runs exceeding their normal window
job.run.duration > job_expected_duration by (job)
# failed runs
count(job.run{result="failed"}) by (job)A job that should have run did not, and because a job that does not run emits nothing, it stays invisible unless you are watching for the absence. Comparing expected schedules against observed runs is the only detector. The causes range from a broken scheduler to a disabled cron entry; downstream data going stale is often the first business symptom.
A job runs far longer than its normal window, so it either collides with the next scheduled run or delays the data everything downstream depends on. Duration against the expected window flags it. Causes include data-volume growth, a missing index, or resource contention; an overrunning nightly job is a freshness incident, not just a slow log line.
A job exits with an error, or completes but processes far fewer records than usual, and nobody notices because nothing paged. Exit status and records-processed together catch it. Treat a failed or suspiciously light run as an incident and re-run or investigate, because downstream systems assume the data is complete.
Starting conditions to tune:
| Condition | Signal | Meaning |
|---|---|---|
| expected run not seen within its window | schedule | A job was missed. |
| run exit status = failed | failure | A job errored. |
| run duration > expected window | overrun | A job is running late. |
| records processed far below normal | integrity | A run may have partially failed. |
Ops Singularity monitors scheduled jobs through TelemetryOps and its DataOps pillar, and resolves run failures, retrying, reallocating or escalating, through governed Action Tickets. Explore TelemetryOps and DataOps.
Wrap each job run in an OpenTelemetry span recording start, end, exit status and attributes, exported over OTLP. On Kubernetes, the k8sobjects receiver also captures Job and CronJob events.
By tracking expected run schedules against observed run spans, so a missed start is flagged, rather than relying on the job to report its own failure.
Bring a real incident. We will show you Sentinel investigate, act and verify end to end, with every action reversible and audited.