Cron is one among the favourites of sysadmins and we usually use cron service to schedule tasks that you want to run in a periodic manner. People usually run cron in a single box. You cannot expect high availability for your cron here because the box can go down anytime and you are in trouble if the cron job is important. In this article, I will explain two methods of cron management, one talks about distributing crons evenly among boxes and the other one talks about distributing crons with high availability.
Method 1: Evenly distribute cron execution among multiple boxes:
You can use this method if you want a cron job to be run in a given number of boxes with the work load distributed evenly.
Assume you have a rsync cron which runs every 5mins and you want it to be evenly distributed among machines box1.test.net and box2.test.net .
Open the crontab with your favourite text editor and add following content. Do this on both boxes.
DISTRIBUTE='if [ ${HOSTNAME%%.*} = box2 ]; then sleep 300; fi'
06,36,46,56 * * * * eval $DISTRIBUTE; /usr/bin/rsync /path/source /path/destination
We have instructed the cron to run 6th, 36th, 46th and 56th minute of every hour. Here eval command with Environment variable DISTRIBUTE do the real work. The IF condition instructs the cron to sleep for 5 mins if the hostname is box2 and cron will execute immediately in box1 because the IF condition fails there. Thus we make sure cron is evenly distributed on both machines.
With this method, we can make sure the cron execution load is evenly distributed, but high availability is not promised here because if any of the box goes down, cron that supposed to run in that box won’t run at all. But don’t worry, method 2 handles high availability.
Method 2: Make sure cron high availability with Flock:
You might have heard about Flock. We can use flock to make sure only one process of an instance is running at time. Flock does it by creating a lock file. In order to use flock in our usecase, you need to attach a common storage to all boxes where you plan to distribute a cron.
Assume you have a rsync cron which runs every 5mins and you want it to be distributed among machines box1.test.net and box2.test.net .
Open the crontab file with your favourite text editor and add following content. Do this on both boxes.
*/5 * * * * /usr/bin/flock -n /common_storage/cron.lock /usr/bin/rsync /path/source /path/destination
Flock initially checks if the lock exists on the file and if it is present, it won’t execute the given command/script. When the cron runs in box1, it creates a lock on file /common_storage/cron.lock . So whenever the cron runs in box2, flock checks for the existence of the lock on the file and if the locks exists, cron(basically the rsync) won’t run in box2. With this setup, we make sure the cron runs every 5mins on any of two boxes and the cron is highly available because it can run with atleast one box.
Note that, the execution load is not evenly distributed among boxes in this method.
Thanks for the time taken to read my blog. Subscribe to this blog so that you don’t miss out anything useful (Checkout Right Sidebar for Facebook follow button and mail subscription form ) . Please also put your thoughts as comments .
