High CPU utilisation for Logs Manager shell script

Question

I have written a shell script to manage the tcpdump pcap logs and syslog files in my Linux board, so as to maintain the disk usage to maximum of 70%.

The script checks for the disk usage every minute and as soon as the disk goes higher than 70%, it deletes all the oldest files that are not in use until the disk usage goes back to 70%.

#!/bin/ash

#Gives the file that is currently in use by the process
logmgr_get_current_file_name() {
    PROC_NAME=$1
    instance=$2

    pid=$(ps | grep $PROC_NAME | grep -v grep | awk 'NR=='$instance'{print $1}')

    fds=$(ls /proc/$pid/fd)

    for fd in $fds;
    do
        last=$fd
    done

    current_file=$(ls /proc/$pid/fd/$last -l | awk 'BEGIN {FS="->"} {printf $2}')
}

#checks whether the file we are deleting is in use by the process 
logmgr_is_file_in_use() {
    num_of_instances=1
    logmgr_get_current_file_name $1 $num_of_instances
    if [ $current_file = $2 ]; then
        return 1
    else
       return  0
    fi
}

# iterate through all possible instances of tcpdump
# for each instance .. get the file in use
# check if [ $file_in_use == $to_delete ] ; then
# return 1
# else
# return 0
logmgr_get_files_in_use() {
    i=1
    is_file_inuse=0

    num_of_instances=$(ps | grep tcpdump| grep -v grep | wc -l)

    while [ $i -le $num_of_instances ];
    do
        logmgr_get_current_file_name $1 $i
        if [ "$current_file" = "$2" ]; then
            is_file_inuse=1
            break
        fi
                i=`expr $i + 1`
        done
        return $is_file_inuse
}

logmgr_delete_oldest() {

    oldest=$(ls -lht $2 | awk 'END{print $9}')
    to_delete=$2$oldest

    if [ "$1" != "tcpdump" ] ; then
        logmgr_is_file_in_use $1 $to_delete
        ret_val=$?
    else
        logmgr_get_files_in_use $1 $to_delete "tcpdump"
        ret_val=$?
    fi

    #if the file is not in use, then delete it
    if [ $ret_val -eq 0 ]; then
        echo "$to_delete deleted"
        rm -fr $to_delete
        sleep $cpu_relaxation_secs
    fi

}


logmgr_get_disk_usage() {
    diskusage=$(df -h $1 | awk 'END {print $5}' | awk 'BEGIN {FS="%"} {print $1}')
    echo $diskusage
}

    ##requires repeated execution######
logmgr_run() {

    sleep_secs=0

    while [ 1 ] ;
    do
        syslog_dir_is_ok=0
        tcpdump_dir_is_ok=0

        syslog_disk_usage=`logmgr_get_disk_usage $syslog_dir`
        tcpdump_disk_usage=`logmgr_get_disk_usage $tcpdump_dir`

        if [ $syslog_disk_usage -le $NORMAL_DISK_USAGE ] ; then
            syslog_dir_is_ok=1
        fi

        if [ $tcpdump_disk_usage -le $NORMAL_DISK_USAGE ] ; then
            tcpdump_dir_is_ok=1
        fi

        if [ $syslog_dir_is_ok -eq 0 ] && [ $tcpdump_dir_is_ok -eq 0 ] ; then
            break
        fi

        if [ $syslog_dir_is_ok -eq 0 ] ; then
            num_files=`ls -lht $syslog_dir | wc -l`
            logmgr_delete_oldest "syslogd" $syslog_dir $num_files
        fi

        if [ $tcpdump_dir_is_ok -eq 0 ] ; then
            num_files=`ls -lht $tcpdump_dir | wc -l`
            logmgr_delete_oldest "tcpdump" $tcpdump_dir $num_files
        fi
    done
}

##Default values##

max_file_safe=100
cpu_relaxation_secs=1
timer_secs=60   # one minute
KEEP_CHECKING=1
NORMAL_DISK_USAGE=20

# read the configuration file
syslog_dir=`uci get syslog.slog.directory`
syslog_dir=`dirname $syslog_dir`"/"`basename $syslog_dir`"/"
tcpdump_dir=`uci get tcpdump.tcpdump.log_dir`
tcpdump_dir=`dirname $tcpdump_dir`"/"`basename $tcpdump_dir`"/"

#while [ $KEEP_CHECKING -eq 1 ]
while :
do
    logmgr_run
    sleep $timer_secs
done

The script is sequential and works perfectly fine but the CPU utilisation gets higher to about 75% while running the script. I tried to remove all the redundancies in script but still not successful in achieving the desired result.

I am sure there are ways with which I can achieve this, but am not sure how. Please suggest some ways to optimise this script and reduce my CPU utilisation.

Answer 1

If your CPU is rising to 75% and staying there (not bouncing up and down), then it suggest that your script is spinning and not hitting any of your sleep statements. There's a few things that may be worth looking into.

• The while loop in logmgr_run keeps going until enough files have been cleared out. If this doesn't happen, then it'll never get to the 60 second sleep.
• logmgr_delete_oldest will only sleep if it deletes a file, which only happens if the file isn't in use.

I haven't tested it, and it has been a while since I did script programming, but it looks to me like your algorithm breaks down if the oldest file in either folder is being used at the same time that you breach the usage threshold. It can't delete the oldest file, so it just keeps trying over and over again without succeeding or sleeping.

Rather than having a loop that keeps trying to remove the last file when the threshold is breached it might be better to calculate the amount of files that need to be removed in order to come back under the threshold and try to remove several files in one go / or to skip files if they are in use and move on to the next file rather than stopping and trying the same file again next time.

It may also be worthwhile extending the cleanup to clear a bit more than you need, so that you don't immediately trigger another cleanup next time you go through the loop. So, for example if you breach 70% usage, clear down the files to 65% or 60% so that there is some growth space.

Your script also has some declared an unused variable, which you should probably either be using (or remove):

max_file_safe=100

Answer 2

The script can and should be simplified.

Use the exit code of commands

The exit code of a function is the exit code of the last statement. With that in mind, this function can be written much simpler:

logmgr_is_file_in_use() {
    num_of_instances=1
    logmgr_get_current_file_name $1 $num_of_instances
    if [ $current_file = $2 ]; then
        return 1
    else
       return  0
    fi
}

Like this:

logmgr_is_file_in_use() {
    num_of_instances=1
    logmgr_get_current_file_name $1 $num_of_instances
    ! [ $current_file = $2 ]
}

Simplify grep -> grep -> awk

awk can do much of what grep can. When you use awk at the end of a pipeline, consider using it more to reduce the number of processes in the pipeline.

Take for example this:

pid=$(ps | grep $PROC_NAME | grep -v grep | awk 'NR=='$instance'{print $1}')

This can be written as:

pid=$(ps | awk -v proc="$PROC_NAME" -v inst=$instance '$0 ~ proc && NR == inst {print $1}')

Avoid ls when you can use globs

This code and the rest that depends on it can be rewritten without an ls:

fds=$(ls /proc/$pid/fd)

for fd in $fds;
do
    last=$fd
done

Like this:

last=
for fd in /proc/$pid/fd/*; do
    last=$fd
done
[ -e "$last" ] || last=

The last line is in case there is no matching file, otherwise the value of last might look like /proc/123/fd/*, which is not a file.

Notice that I emptied last before the loop, to avoid potential bugs due to a pre-existing value.

Never ever parse the output of ls

This is a very clumsy way to find the target of a symbolic link.

current_file=$(ls /proc/$pid/fd/$last -l | awk 'BEGIN {FS="->"} {printf $2}')

It seems you're using the ash shell, which should have a readlink command that can do this much cleaner:

current_file=$(readlink /proc/$pid/fd/$last)

Use consistent techniques

I can see multiple equivalent techniques to accomplish the same thing:

• Execute sub-shell using $(...) or `...`
• Infinite loop using while :; do or while [ 1 ]; do

Be consistent. Use the same technique throughout, and the code will be easier. Of course, pick the better technique. In these examples, the first of the pairs are the better.