Cron job monitoring: catch silent failures with a heartbeat
In short
Cron jobs fail silently because cron only reports output by local mail, which most servers never deliver, and because a job that does not start cannot report anything. The reliable fix is a heartbeat: the job calls a secret URL after it succeeds, and a monitoring service raises an alert when the call is overdue. One curl command at the end of the job is enough.
Cron jobs fail silently because cron has no real way to tell you. It can send the output of a job as local mail, and on most servers that mail goes nowhere. A job that never starts produces no output at all. The dependable way to notice is to turn the logic around: the job reports in after every successful run, and a monitoring service alerts you when that report is overdue. This is called a heartbeat or dead man's switch, and it takes one curl command.
Why do cron jobs fail without anyone noticing?
There are three different ways a job can go wrong, and classic error handling covers only one of them.
The job runs and fails. The script exits with an error. Cron collects whatever it printed and tries to mail it to the owner of the crontab or the address in MAILTO. Without a configured mail transfer agent, the output is discarded or ends up in a local mailbox nobody reads.
The job does not start. No output, no mail, no log line in your application. Common reasons:
- The cron service is not running, or the server was down at the scheduled time.
- The crontab was overwritten during a deployment or a server move.
- The script path or its permissions changed.
- The job works in your shell but not in cron, because cron runs with a minimal environment and a short
PATH. - A percent sign in the crontab line. Cron treats an unescaped
%as a line break, which breaks commands such asdate +%F. Write\%instead.
The job starts and never finishes. It waits for a lock, a network mount or a remote host. The next run piles up behind it.
A check of the kind "alert me when an error occurs" can only catch the first case. The other two are the absence of an event, and you can only detect an absence from the outside.
How do I check whether a cron job ran?
For a quick look, cron logs every start. On Debian and Ubuntu:
journalctl -u cron --since today | grep CMDNov 18 02:30:01 web-01 CRON[41822]: (root) CMD (/usr/local/bin/backup.sh)On systems with rsyslog the same lines are in /var/log/syslog. On the RHEL family the service is called crond and logs to /var/log/cron. For systemd timers:
systemctl list-timers
systemctl status backup.serviceNote what this log proves: the job was started. It says nothing about whether it succeeded. And reading logs by hand works for an investigation, not as monitoring.
How does the heartbeat pattern work?
You create a check in a monitoring service and tell it the schedule: for example "reports every 24 hours, with one hour of grace". The service gives you a secret URL. The job calls this URL when it has finished successfully. As long as the calls arrive on time, nothing happens. When a call is overdue, the service opens an incident and notifies you.
This covers all three failure cases with the same mechanism. A failed job does not call. A job that never started does not call. A hanging job does not call in time. It also covers causes outside the script: a stopped cron service, a server that is switched off, a firewall change.
What does the curl call look like?
The following examples use a placeholder URL. Your monitoring service gives you the real one.
In a crontab, chain the call with && so that it only runs when the job exits with code 0:
30 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://monitoring.example/ping/YOUR-TOKENThe options are chosen deliberately:
-fmakes curl return an error on HTTP status 400 and above, instead of treating an error page as success.-sShides the progress display but still prints errors.-m 10limits the whole call to ten seconds, so that a network problem cannot make the job hang.--retry 5repeats the call on temporary failures.-o /dev/nulldiscards the response body.
Inside a script, put the call at the very end and make sure the script stops at the first error:
#!/bin/bash
set -euo pipefail
mysqldump --single-transaction shop | gzip > /var/backups/db/shop-$(date +%F).sql.gz
rsync -a /var/backups/db/ backup-host:/srv/backups/web-01/
curl -fsS -m 10 --retry 5 -o /dev/null https://monitoring.example/ping/YOUR-TOKENset -e ends the script when a command fails, and pipefail makes a pipeline fail when any part of it fails. Without pipefail, a failed mysqldump piped into gzip counts as success, and the heartbeat would report a backup that is an empty file.
For a systemd timer, add the call to the service unit:
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 5 -o /dev/null https://monitoring.example/ping/YOUR-TOKENsystemd runs ExecStartPost only after ExecStart has finished successfully.
Many heartbeat services also accept a start signal and an explicit failure signal. With a start signal the service can measure the run time and notice a job that started but never finished. With a failure signal you hear about a failed run immediately instead of after the grace period.
How do I choose interval and grace period?
The interval is the schedule of the job. The grace period is how late the report may be before it counts as missed. It has to cover the run time of the job and its normal variation.
| Job | Interval | Typical grace |
|---|---|---|
| Queue worker check every 5 minutes | 5 minutes | 5 minutes |
| Hourly import | 1 hour | 15 minutes |
| Nightly backup, runs 20 to 40 minutes | 1 day | 1 hour |
| Weekly report | 7 days | a few hours |
Too short a grace period produces false alarms and teaches everyone to ignore the check. Too long, and you learn about the missed backup a day later. Start generously, look at real run times for two weeks, then tighten.
Common mistakes
- A separate crontab line for the ping. A line that pings at 03:00 while the backup runs at 02:30 proves that cron works, not that the backup did. The ping must depend on the exit code of the job.
- Using a semicolon instead of
&&. With;the ping is sent even after a failure. - Pinging at the start only. That reports "started", not "finished".
- No timeout on the curl call. Without
-m, an unreachable monitoring service can block the job. - Treating the URL as public. Anyone who knows it can report success. Keep it out of public repositories and tickets.
- Checking the run but not the result. A heartbeat says the script exited with 0. Whether the backup file is complete and restorable is a second question.
- Overlapping runs. If a job can take longer than its interval, start it through
flock -n /run/lock/jobname.lock /usr/local/bin/job.shso that runs do not pile up.
Scheduled jobs that deserve a heartbeat are not only backups. Certificate renewal is a classic: when the renewal timer stops working, nothing happens for weeks and then the site shows a browser warning. More on that in TLS certificate expiry monitoring. Log rotation and cleanup jobs are another case, since a cleanup that stops quietly is a common cause of a full disk.
Checklist
- List all scheduled jobs:
crontab -lfor each user,/etc/cron.d/,systemctl list-timers. - Mark the ones whose failure would hurt: backups, renewals, imports, invoicing, cleanup.
- Create one heartbeat check per job, with interval and grace period.
- Add the curl call with
&&, or asExecStartPost, and useset -euo pipefailin scripts. - Test the alert: comment the job out for one cycle and confirm that the notification arrives.
- Decide who is notified and when, so that a missed nightly job does not wake someone needlessly. See alert fatigue, quiet hours and escalation.
How Lynx Heartbeat helps
Lynx Heartbeat has heartbeat monitors for exactly this pattern. Each monitor has a secret ping URL and a schedule, given either as an interval or as a cron expression, plus a grace period; when the report is overdue, the monitor opens an incident and notifies you through the same channels as every other check. Besides the plain success ping, a job can report its start by appending /start to the URL, a failure with /fail, or simply its exit code, where 0 counts as success and anything else as failure. The monitor page shows ready-made examples for crontab, shell scripts and systemd units with your real URL. The other check types are listed on the features page.
Frequently asked questions
Why did nobody notice that my cron job stopped running?
Cron reports problems only by sending the output of a job as local mail. Most servers have no working mail delivery for this, so the message is lost. And if the job never starts, for example because the cron service is stopped or the crontab was replaced, there is no output at all.
What is a heartbeat or dead man's switch for cron jobs?
The job reports in after each successful run by calling a secret URL. A monitoring service knows the schedule and raises an alert when the call does not arrive within the expected interval plus a grace period. It detects jobs that fail, jobs that hang and jobs that never start.
How do I add a heartbeat to an existing cron job?
Append a curl call with two ampersands to the crontab line, for example: backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null followed by the ping URL. The call is only made when the job exits with code 0.
How long should the grace period be?
Long enough to cover the normal variation in run time, short enough that you still hear about a failure in time. For a nightly backup that takes 20 to 40 minutes, one hour is a sensible start. Measure real run times for a week or two and adjust.
Does this also work with systemd timers?
Yes. Add the curl call as ExecStartPost to the service unit. systemd runs it only when the main command has succeeded. Alternatively, use an OnFailure unit to report failures actively.