2
\$\begingroup\$

My Python-based web server (Tornado module) is pretty complex and keeps going down (seems like usually in the middle of the night), so here's a bash script that kills the process, restarts and emails and txt messages me:

#!/bin/bash
NOTIFYEMAIL=<my_email_address>
SMSEMAIL=<cell_phone_number>@<carrier> # ie. pm.sprint.com
SENDEREMAIL=admin@loclahost
SERVER=http://the_website.com/
PAUSE=60
FAILED=0
DEBUG=0

while true 
do
/usr/bin/curl -sSf --max-time 10 $SERVER > /dev/null 2>&1
CS=$?
# For debugging purposes
if [ $DEBUG -eq 1 ]
then
    echo "STATUS = $CS"
    echo "FAILED = $FAILED"
    if [ $CS -ne 0 ]
    then
    echo "$SERVER is down"

    elif [ $CS -eq 0 ]
    then
    echo "$SERVER is up"
    fi
fi

# If the server is down and no alert is sent - alert
if [ $CS -ne 0 ] && [ $FAILED -eq 0 ]
then
    PID=`ps -eaf | grep 'python -m fore.server' | grep -v grep | awk '{print $2}'`
    FAILED=1
    if [ $DEBUG -eq 1 ]
    then
    echo "$SERVER failed"
    if [[ "" !=  "$PID" ]]
    then
        echo "killing $PID"
        kill -9 $PID
        echo "Starting the server."
        python -m fore.server
    else
        echo "No Python Process Running Locally"
    fi
    fi
    if [ $DEBUG = 0 ]
    then
    echo "$SERVER failed"
    PID=`ps -eaf | grep 'python -m fore.server' | grep -v grep | awk '{print $2}'`
    if [[ "" !=  "$PID" ]]
    then
        kill -9 $PID
        echo "$SERVER went down $(date)" | /usr/bin/mail -s "$SERVER went down" "$SENDEREMAIL" "$SMSEMAIL" 
        echo "$SERVER went down $(date)" | /usr/bin/mail -s "$SERVER went down" "$SENDEREMAIL" "$NOTIFYEMAIL"
        python -m fore.server
    else
        echo "No Python Process Running Locally"
    fi 
    fi

# If the server is back up and no alert is sent - alert
elif [ $CS -eq 0 ] && [ $FAILED -eq 1 ]
then
    FAILED=0
    if [ $DEBUG -eq 1 ]
    then
    echo "$SERVER is back up"
    fi
    if [ $DEBUG = 0 ]
    then
    echo "$SERVER is back up $(date)" | /usr/bin/mail -s "$SERVER is back up again" "$SENDEREMAIL" "$SMSEMAIL"
    echo "$SERVER is back up $(date)" | /usr/bin/mail -s "$SERVER is back up again" "$SENDEREMAIL" "$NOTIFYEMAIL"
    fi
fi
sleep $PAUSE
done

I have to execute my server with sudo in order to use port :80 so I am executing the script like:

sudo ./outage.sh

Of course, warnings, suggestions, improvements are welcome.

One improvement I'm interested in implementing is to iterate through a list of servers in order to keep track of multiple web sites.

\$\endgroup\$
  • \$\begingroup\$ Well I've already found one likely flaw with this script which is that when the server gets restarted, I'm not sure what happens with this one - i think it stops running or at least is blocked, as opposed to spawning a separate process. \$\endgroup\$ – MikeiLL Apr 15 '15 at 19:17
1
\$\begingroup\$

Indent code blocks

To improve readability, it's good to indent code blocks in loops and if statements. For example:

while true
do
    /usr/bin/curl -sSf --max-time 10 $SERVER > /dev/null 2>&1
    CS=$?

You indented some if statements, but not all. Indent all.

Replace elif with else if possible

In these conditions:

    if [ $CS -ne 0 ]
    then
    echo "$SERVER is down"

    elif [ $CS -eq 0 ]
    then
    echo "$SERVER is up"

If $CS is not-not-zero, then when can it be? Zero, of course! So the elif can be a simple else.

Use $(...) instead of `...`

`...` is old-fashioned and error prone. Use $(...) instead.

ps ... | grep ... | grep -v grep

Instead of:

ps -eaf | grep 'python -m fore.server' | grep -v grep

A common workaround is:

ps -eaf | grep '[p]ython -m fore.server'

This works, because this way in the output of ps, the process python -m fore.server will still match, but grep '[p]ython -m fore.server' will not match.

However, in this particular example you should go a bit further, see the next point.

Look suspiciously at grep in a pipeline ending with awk

awk is fast and powerful, and very often can easily do grep's job too when it's in the same pipeline.

For example instead of this:

ps -eaf | grep 'python -m fore.server' | grep -v grep | awk '{print $2}'

You can rewrite the whole thing using awk alone:

ps -eaf | awk '/python -m fore.server/ {print $2}'

This is not only shorter and less troublesome (no need for the grep [p]ython trick), and there are less processes executed in the pipeline.

Don't repeat yourself

You do some things twice, with identical or very similar commands:

  • Finding the PID of the Python process
  • Sending an email with the message that the server is down
  • Sending an email with the message that the server is back up

It would be better to move common operations to helper functions.

Other simplifications

Instead of this:

    if [[ "" !=  "$PID" ]]

You can write as:

    if [[ $PID ]]
\$\endgroup\$
  • 1
    \$\begingroup\$ wow. that's a lot to take in at my level of bash. have barely cracked the surface of awk yet. \$\endgroup\$ – MikeiLL Apr 30 '15 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.