Skip to content

Cron job monitoring: catch silent failures with a heartbeat

Cron jobs fail silently because cron only reports output by local mail, which most servers never deliver, and because a job that does not start cannot report anything. The reliable fix is a heartbeat: the job calls a secret URL after it succeeds, and a monitoring service raises an alert when the call is overdue. One curl command at the end of the job is enough.

Cron jobs fail silently because cron has no real way to tell you. It can send the output of a job as local mail, and on most servers that mail goes nowhere. A job that never starts produces no output at all. The dependable way to notice is to turn the logic around: the job reports in after every successful run, and a monitoring service alerts you when that report is overdue. This is called a heartbeat or dead man's switch, and it takes one curl command.

Why do cron jobs fail without anyone noticing?

There are three different ways a job can go wrong, and classic error handling covers only one of them.

The job runs and fails. The script exits with an error. Cron collects whatever it printed and tries to mail it to the owner of the crontab or the address in MAILTO. Without a configured mail transfer agent, the output is discarded or ends up in a local mailbox nobody reads.

The job does not start. No output, no mail, no log line in your application. Common reasons:

  • The cron service is not running, or the server was down at the scheduled time.
  • The crontab was overwritten during a deployment or a server move.
  • The script path or its permissions changed.
  • The job works in your shell but not in cron, because cron runs with a minimal environment and a short PATH.
  • A percent sign in the crontab line. Cron treats an unescaped % as a line break, which breaks commands such as date +%F. Write \% instead.

The job starts and never finishes. It waits for a lock, a network mount or a remote host. The next run piles up behind it.

A check of the kind "alert me when an error occurs" can only catch the first case. The other two are the absence of an event, and you can only detect an absence from the outside.

How do I check whether a cron job ran?

For a quick look, cron logs every start. On Debian and Ubuntu:

journalctl -u cron --since today | grep CMD
Nov 18 02:30:01 web-01 CRON[41822]: (root) CMD (/usr/local/bin/backup.sh)

On systems with rsyslog the same lines are in /var/log/syslog. On the RHEL family the service is called crond and logs to /var/log/cron. For systemd timers:

systemctl list-timers
systemctl status backup.service

Note what this log proves: the job was started. It says nothing about whether it succeeded. And reading logs by hand works for an investigation, not as monitoring.

How does the heartbeat pattern work?

You create a check in a monitoring service and tell it the schedule: for example "reports every 24 hours, with one hour of grace". The service gives you a secret URL. The job calls this URL when it has finished successfully. As long as the calls arrive on time, nothing happens. When a call is overdue, the service opens an incident and notifies you.

This covers all three failure cases with the same mechanism. A failed job does not call. A job that never started does not call. A hanging job does not call in time. It also covers causes outside the script: a stopped cron service, a server that is switched off, a firewall change.

What does the curl call look like?

The following examples use a placeholder URL. Your monitoring service gives you the real one.

In a crontab, chain the call with && so that it only runs when the job exits with code 0:

30 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://monitoring.example/ping/YOUR-TOKEN

The options are chosen deliberately:

  • -f makes curl return an error on HTTP status 400 and above, instead of treating an error page as success.
  • -sS hides the progress display but still prints errors.
  • -m 10 limits the whole call to ten seconds, so that a network problem cannot make the job hang.
  • --retry 5 repeats the call on temporary failures.
  • -o /dev/null discards the response body.

Inside a script, put the call at the very end and make sure the script stops at the first error:

#!/bin/bash
set -euo pipefail

mysqldump --single-transaction shop | gzip > /var/backups/db/shop-$(date +%F).sql.gz
rsync -a /var/backups/db/ backup-host:/srv/backups/web-01/

curl -fsS -m 10 --retry 5 -o /dev/null https://monitoring.example/ping/YOUR-TOKEN

set -e ends the script when a command fails, and pipefail makes a pipeline fail when any part of it fails. Without pipefail, a failed mysqldump piped into gzip counts as success, and the heartbeat would report a backup that is an empty file.

For a systemd timer, add the call to the service unit:

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 5 -o /dev/null https://monitoring.example/ping/YOUR-TOKEN

systemd runs ExecStartPost only after ExecStart has finished successfully.

Many heartbeat services also accept a start signal and an explicit failure signal. With a start signal the service can measure the run time and notice a job that started but never finished. With a failure signal you hear about a failed run immediately instead of after the grace period.

How do I choose interval and grace period?

The interval is the schedule of the job. The grace period is how late the report may be before it counts as missed. It has to cover the run time of the job and its normal variation.

JobIntervalTypical grace
Queue worker check every 5 minutes5 minutes5 minutes
Hourly import1 hour15 minutes
Nightly backup, runs 20 to 40 minutes1 day1 hour
Weekly report7 daysa few hours

Too short a grace period produces false alarms and teaches everyone to ignore the check. Too long, and you learn about the missed backup a day later. Start generously, look at real run times for two weeks, then tighten.

Common mistakes

  • A separate crontab line for the ping. A line that pings at 03:00 while the backup runs at 02:30 proves that cron works, not that the backup did. The ping must depend on the exit code of the job.
  • Using a semicolon instead of &&. With ; the ping is sent even after a failure.
  • Pinging at the start only. That reports "started", not "finished".
  • No timeout on the curl call. Without -m, an unreachable monitoring service can block the job.
  • Treating the URL as public. Anyone who knows it can report success. Keep it out of public repositories and tickets.
  • Checking the run but not the result. A heartbeat says the script exited with 0. Whether the backup file is complete and restorable is a second question.
  • Overlapping runs. If a job can take longer than its interval, start it through flock -n /run/lock/jobname.lock /usr/local/bin/job.sh so that runs do not pile up.

Scheduled jobs that deserve a heartbeat are not only backups. Certificate renewal is a classic: when the renewal timer stops working, nothing happens for weeks and then the site shows a browser warning. More on that in TLS certificate expiry monitoring. Log rotation and cleanup jobs are another case, since a cleanup that stops quietly is a common cause of a full disk.

Checklist

  1. List all scheduled jobs: crontab -l for each user, /etc/cron.d/, systemctl list-timers.
  2. Mark the ones whose failure would hurt: backups, renewals, imports, invoicing, cleanup.
  3. Create one heartbeat check per job, with interval and grace period.
  4. Add the curl call with &&, or as ExecStartPost, and use set -euo pipefail in scripts.
  5. Test the alert: comment the job out for one cycle and confirm that the notification arrives.
  6. Decide who is notified and when, so that a missed nightly job does not wake someone needlessly. See alert fatigue, quiet hours and escalation.

How Lynx Heartbeat helps

Lynx Heartbeat has heartbeat monitors for exactly this pattern. Each monitor has a secret ping URL and a schedule, given either as an interval or as a cron expression, plus a grace period; when the report is overdue, the monitor opens an incident and notifies you through the same channels as every other check. Besides the plain success ping, a job can report its start by appending /start to the URL, a failure with /fail, or simply its exit code, where 0 counts as success and anything else as failure. The monitor page shows ready-made examples for crontab, shell scripts and systemd units with your real URL. The other check types are listed on the features page.

Frequently asked questions

Why did nobody notice that my cron job stopped running?

Cron reports problems only by sending the output of a job as local mail. Most servers have no working mail delivery for this, so the message is lost. And if the job never starts, for example because the cron service is stopped or the crontab was replaced, there is no output at all.

What is a heartbeat or dead man's switch for cron jobs?

The job reports in after each successful run by calling a secret URL. A monitoring service knows the schedule and raises an alert when the call does not arrive within the expected interval plus a grace period. It detects jobs that fail, jobs that hang and jobs that never start.

How do I add a heartbeat to an existing cron job?

Append a curl call with two ampersands to the crontab line, for example: backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null followed by the ping URL. The call is only made when the job exits with code 0.

How long should the grace period be?

Long enough to cover the normal variation in run time, short enough that you still hear about a failure in time. For a nightly backup that takes 20 to 40 minutes, one hour is a sensible start. Measure real run times for a week or two and adjust.

Does this also work with systemd timers?

Yes. Add the curl call as ExecStartPost to the service unit. systemd runs it only when the main command has succeeded. Alternatively, use an OnFailure unit to report failures actively.

Operations

Related articles

All articles