Help / Monitors
Heartbeat monitor (cron jobs)
A heartbeat monitor works the other way round from the others. Instead of HonkWatch checking something, your job calls a web address each time it runs. If the call does not arrive on time, you get an alert. Use it for backups, cron jobs, scheduled tasks and scripts that should run on their own.
What you need first
- A job that runs on a schedule, and a way to add one line to it.
- A tool on that machine that can open a web address, such as
curl,wgetor PowerShell. - The machine must be able to reach the internet.
Set it up
- Open Monitors and press Add monitor.
- Choose Heartbeat (cron job).
- Give it a Friendly name, such as "Nightly database backup".
- In the list under Friendly name, choose how often the job runs. On a new form it is labelled Check every; on a saved heartbeat monitor it reads Expected every.
- In Grace period (seconds), choose how late the job may be before you are told. The default is 60 seconds. You can choose from 0 to 86400 (one day).
- Look over Where alerts go and press Start monitoring.
- The monitor page opens. Under Heartbeat URL it shows your address, as a ready-made
curlline. Copy it.
Treat the heartbeat URL like a password. Anyone who has it can send heartbeats for this monitor. Team members with the Viewer role do not see it.
Jobs that run less often than every 6 hours
The longest choice in Expected every is 6 hours. For a job that runs once a day, choose 6 hours and add the rest as grace. The alert comes when the time since the last call is more than the two added together. For example, 6 hours plus a grace period of 68400 seconds (19 hours) gives 25 hours, which allows a daily job to run up to an hour late.
Add the call to your job
Put the call at the end of the job, so it only runs when the job has finished. The address accepts GET, POST and HEAD requests and answers ok.
On Linux or macOS, in a crontab, run the call only when the job succeeds:
0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 YOUR-HEARTBEAT-URL
On Windows, add this as the last step of the scheduled task or script:
Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 YOUR-HEARTBEAT-URL
Replace YOUR-HEARTBEAT-URL with the address from the monitor page.
How it decides
- Until the first call arrives, the monitor shows "Waiting for the first check".
- Each call marks the monitor up and starts the countdown again.
- If no call arrives within Expected every plus the grace period, the monitor goes down with the reason "No heartbeat received" and a down alert goes out. This also applies to a new monitor that never gets its first call.
- The next call brings it back up and sends a recovery message, unless you turned that off under More settings.
- Calls that come within 5 seconds of each other count once in the history, so a script that calls in a loop does not fill it up.
- Pausing the monitor stops the countdown. When you resume, it starts again from that moment.
Under More settings you can also make the down alert repeat until someone acknowledges the incident.
Send a test
- Run the
curlline from the monitor page by hand, in a terminal on the machine that runs the job. It should printok. - Reload the monitor page. It should say Working, and Last ping should show a moment ago.
- To check that alerts reach you, open Alerts and press Send test on each channel.
When it does not work
- The call prints "unknown heartbeat". The address is wrong or incomplete, or the monitor was deleted or changed to another type. Copy the address again from the monitor page.
- The call works by hand but not from the job. Scheduled jobs often run with a shorter path or as another user. Use the full path to
curl, and check the job's own log for errors. - The call times out or cannot connect. The machine cannot reach the internet, or a firewall or proxy blocks outgoing requests.
- You get alerts although the job ran. The job may take longer than you allowed. Increase the grace period. If the call sits behind
&&, a job that finishes with an error does not call, which is what you want. - You never get an alert. Check that a channel has Down ticked on the Alerts page, and see I did not get an alert.
Still stuck? Email [email protected] and say which guide you were following.