Linux: Monitor a Service with a Watchdog Script

Old Unix hands already know this, but new Unix (Linux) users may be asking, ‘What is a “watchdog script”?’ Basically it is a bash or other script that is run via cron periodically to check on a persistent service. The watchdog script takes actions based on the state of the service it monitors.
There are other examples of watchdog scripts on the internet. Just search for them using your favorite search engine to see them. Following is a watchdog script we created recently for a client to monitor an e-mail to pager system my company wrote for the client. Here is the script (with sensitive bits changed to protect the innocent):

#!/bin/bash
#
# watchdog
#
# Run as a cron job to keep an eye on what_to_monitor which should always
# be running. Restart what_to_monitor and send notification as needed.
#
# This needs to be run as root or a user that can start system services.
#
# Revisions: 0.1 (20100506), 0.2 (20100507)

NAME=what_to_monitor
START=/full/path/to/$NAME
NOTIFY=person1email
NOTIFYCC=person2email
GREP=/bin/grep
PS=/bin/ps
NOP=/bin/true
DATE=/bin/date
MAIL=/bin/mail
RM=/bin/rm

$PS -ef|$GREP -v grep|$GREP $NAME >/dev/null 2>&1
case "$?" in
   0)
   # It is running in this case so we do nothing.
   $NOP
   ;;
   1)
   echo "$NAME is NOT RUNNING. Starting $NAME and sending notices."
   $START 2>&1 >/dev/null &
   NOTICE=/tmp/watchdog.txt
   echo "$NAME was not running and was started on `$DATE`" > $NOTICE
   $MAIL -n -s "watchdog notice" -c $NOTIFYCC $NOTIFY < $NOTICE
   $RM -f $NOTICE
   ;;
esac

exit

In case you are a new Linux administrator and are virgin to all things Unix-ish we will explain what this script does.
First of all, if you want to run a script unattended in cron then the first line with “#!”, called a “shebang”, tells whatever is calling the script what to use for processing the script. In this case we want to use bash so the line is “#!/bin/bash” for that. If this were a Perl script then the shebang line may look like “#!/usr/bin/perl”, depending on where the Perl executable resides on your system.
Following the shebang line are several lines of comments, which should be self explanatory. Then the variables used in our script are assigned. These too should be self explanatory. If not please post a comment to ask about them.
The “NAME=what_to_monitor” line is quite important for our purposes. This is the actual name of the program or script that would show up in a process list. We use that in the script to check if that shows up in the process list in the line:

$PS -ef|$GREP -v grep|$GREP $NAME >/dev/null 2>&1

Yes, we could actually try to find a process ID number (PID) for the application we want to monitor. However, as long as the application has a unique name in the process list the method used here will work just fine. There is more we could do to see if the application is hung even though it shows up in the process list. In the case of this particular process we are monitoring it will not hang but may die for some reason or another. If it dies then it will immediately, or nearly immediately, disappear from the process list.
The “$START 2>&1 >/dev/null &” line in our watchdog actually starts our process using the original process script itself from the service’s home directory. This could instead call the “/etc/rc.d/init.d/startupscript” for the script or program that is run as a service. The NAME variable, START variable and line to start the service would then look something like:

NAME=startupscript
START=/etc/rc.d/init.d/$NAME
…
$START start 2>&1 >/dev/null &

Presuming the startupscript uses the word “start” to start the service.
Once we have our script written we want to use it in cron. We use root’s cron for this but one could use any user that has the ability to (re)start system services. We save the watchdog script under /root/bin/watchdog, set it to be executable with “chmod 700 /root/bin/watchdog” and call it from cron using the following crontab line:

* * * * * /root/bin/watchdog

This causes the watchdog to run every minute so it checks the service as often as possible. One can modify the crontab line to run the watchdog whenever one needs it to run. But for persistent services that need to be running we always use a once per minute cron job for our watchdog scripts.
In this script we redirect the majority of our output to /dev/null because we do not want to inundate root’s, or the calling user’s, e-mail with cron job messages every minute. The default in cron is to mail the output from cron jobs to the calling user’s e-mail account. We do want to notify someone when a problem occurs causing our watchdog to trigger. So the NOTIFY and NOTIFYCC variables are set to the local or remote e-mail addresses of the people who need to be notified. Then these lines handle the notification message:

NOTICE=/tmp/watchdog.txt
echo “$NAME was not running and was started on `$DATE`” > $NOTICE
$MAIL -n -s “watchdog notice” -c $NOTIFYCC $NOTIFY < $NOTICE

Please feel free to post comments with pointers to other watchdog scripts or to “fine tune” what is shown here. Questions are also welcome.
http://gengwg.blogspot.com/