Cron jobs help us to run tasks on an automated schedule, however they're often forgotten about once they're set. If you're not monitoring the status of your cron jobs regularly you won't know if they are completing or failing which could lead to issues within your infrastructure.
Server Density cannot monitor cron jobs natively, however we have prepared a wrapper script for you to allow you to capture the exit status of your cron job as the metric cron.job.exit_status, tagged with the job command, and post it back as a metric to your Server Density account. This allows you alert on the exit status, and graph the historical exit status of your cron jobs over time in your account.
Wrapper Script
The script can be downloaded from the Server Density archive. Once downloaded ensure that it can be executed with chmod:
chmod +x sd-cron-wrapper.sh
Then execute your cron job commands with the script to ensure that it can execute your job and return the status. To do this you will need your account name and an agent key of the device you want this metric:
./sd-cron-wrapper.sh -a example -k 1e510f9451850ebab6f7f4c950db59a2 -c '/full/path/to/mycommand with -args'
You should see the script output something similar to the following:
$ ./sd-cron-wrapper.sh -a example -k 1e510f9451850ebab6f7f4c950db59a2 -c date Mon Nov 11 18:24:47 GMT 2019: Starting job Mon Nov 11 18:24:47 GMT 2019: Executing 'date' Mon Nov 11 18:24:47 GMT 2019: date output: Mon Nov 11 18:24:47 GMT 2019 Mon Nov 11 18:24:47 GMT 2019: date exited with 0 Mon Nov 11 18:24:47 GMT 2019: Posting back status to Server Density Mon Nov 11 18:24:47 GMT 2019: SD Cron Wrapper Success
And when looking at your device in your account you should now be able to see the cron.job.exit_status metric.
Customizing Tags
By default the script will tag your metric with command:$JOB_COMMAND. For example, if you execute pgdump with the script the cron.job.exit_status metric will be tagged with command:pgdump. You can override this and set your own tags by using the -t option.
If you provide a single string, the tag will become command:$STRING. Eg;
./sd-cron-wrapper.sh -a example -k 1e510f9451850ebab6f7f4c950db59a2 -c '/full/path/to/mycommand with -args' -t mytag
Would mean that the cron.job.exit_status metric will be tagged with command:mytag.
If you provide a key:value pair, that will become the tag. Eg;
./sd-cron-wrapper.sh -a example -k 1e510f9451850ebab6f7f4c950db59a2 -c '/full/path/to/mycommand with -args' -t mykey:myvalue
Would mean that the cron.job.exit_status metric will be tagged with mykey:myvalue.
Configuring Cron
Now you need to replace your current cron jobs with the wrapper script:
* * * * * /sd-cron-wrapper.sh -a example -k 1e510f9451850ebab6f7f4c950db59a2 -c '/full/path/to/mycommand with -args' >>/var/log/mycommand.cron.log 2>&1"
The above would run the wrapper script every minute, logging the output of the script to /var/log/mycommand.cron.log
There's multiple resources online to help you build crontab expressions.
Configuring Alerts
Once your cron jobs are replaced with the wrapper script you can create alerts for the cron.job.exit_status metric. We recommend creating an alert on tag wildcards, notifying you when the metric != 0, as per the example below:
Monitoring crond
As well as monitoring your cron jobs you will want to monitor the crond process to ensure it is running too, as without crond none of your cron jobs will execute. You can do so by creating the following alert.
Need Help?
If you need any further help or assistance with monitoring cron, or if you have any issues with the script, please get in touch with us via hello@serverdensity.com
Comments