BASH script to monitor subprocess and throttle it for CPU temperature control

Question

I need to run CPU-intensive tasks on a very old machine with overheating issues. The script below will monitor temperature and pause the jobs when it gets too high, continuing them when it's back to normal.

The actual commands run are, of course, not included, since they are irrelevant to the question.

I am looking for hidden traps I may have set in my code (listed at the bottom), and for other things I have done incorrectly. Aside from special characters in the commands and arguments that are run, which are hand created so I can control that risk, what traps or "got-ya's" have I unknowingly set into the code? What ways are there for making this more error-proof, or better in other ways?

For the timing function I know I could have used the

time { command ...; command ...; }

construct, but I was more interested in the time spent by the machine (and previously, by me in the chair) than in the CPU time involved.

The Script:

The code comments should explain what it does, as well as why I did some of it the way I did.

#!/bin/bash

# Build my time reporting function
function report {
    # Get the current time, do the math, report the results.
    end_time=$(date +%s);

    # The time used for the last run process
    proc_time=$(echo "$end_time"-"$start_time" | bc);
    echo " ******* Processing time: $(date -u -d @${proc_time} +%T)";

    # The cummulative time for all processes so far 
    run_time=$(echo "$end_time"-"$launch_time" | bc);
    echo " ******* Running time: $(date -u -d @${run_time} +%T)";
}

# The high and low temperatures to monitor for. Processing is paused
# once the high temp is reached, and will not resume again until the
# low temp is reached.

# My system recovers to 60°C reasonably quick (idle is around 45°C)
temp_lo=60;

# My system dies at about 115°C - since 100°C is normal, suggests my
# sensors are not accurate, but I work with what I have.
# 20°C margin allows for delay in the detection of the high temp, and
# delay in the process pausing, while still keeping temp under danger
# zone. Also allows for when Core0 is rising faster than Core1. They
# seem to take turns being the leader, but seldom more than 5-10°C 
# difference.
temp_hi=95;

# The routine to read the CPU temp with lm sensors. Could be coded
# inline in the watch_child function, but that means placing it in
# three places, and if the grep/sed needs adjusting, then I have to
# remember to change _all_ three, and not make any typos. This cuts
# my chance of errors to a third.
function get_temp {
    # the grep and/or sed may need changing for other sensor output
    # on different systems
    sensors | grep 'Core1' | sed -e 's/.*: \+\([+-][0-9.]\+\)°C.*$/0\1/'
}

# Routine to monitor the CPU temp, pausing the processing as needed
# to remain in the 'safe' range for processor temperature.
function watch_child {
    # argument should be the PID of the backgrounded process
    childd=$1;
    # pre-load the CPU temp
    temp=$(get_temp);
    # As long as the backgrounded process is still running
    while [ -e /proc/$childd ]; do
        # Monitor the process, for still running, and the temp, still
        # safe
        while [ -e /proc/$childd ] && [ $(echo "$temp < $temp_hi" | bc) = 1 ]; do
            # wait a spell
            sleep 5;
            # re-load the temp for a re-check
            temp=$(get_temp);
        done
        # If the process is still running, then it was over-temp that
        # caused the while loop to end
        if [ -e /proc/$childd ]; then
            # Tell the process to take a break
            kill -SIGSTOP "$childd";
        fi
        # Drops through here if the process has ended, otherwise,
        # monitor the temp for a restart
        while [ -e /proc/$childd ] && [ $(echo "$temp > $temp_lo" | bc) = 1 ]; do
            # wait a spell
            sleep 5;
            # re-load the temp for a re-check
            temp=$(get_temp);
        done
        # Drop through here if the process has ended.
        if [ -e /proc/$childd ]; then
            # Otherwise, tell the process that the break is over.
            kill -SIGCONT "$childd";
        fi
    done
    # Only get this far once the process has ended.
    # In the rare case of the process never waking up, the outer while
    # loop will run infinitely!
    # Human monitoring still required!
}

# Start the timer for cumulative run time reports 
launch_time=$(date +%s);

echo "********* The step to perform.";
# Start the timer for this process
start_time=$(date +%s);
# Launch the dangerous process in the background
my_long_running_command arg1 arg2 &
# Capture its PID
child=$!;
# Block, with temp throttling, until this process is done
watch_child $child;
report;

echo "********* The next step to perform.";
# Start the timer for the next process
start_time=$(date +%s);
# Launch the dangerous process in the background
another_long_running_command arg1 arg2 &
# Capture its PID
child=$!;
# Block, with temp throttling, until this process is done
watch_child $child;
report;

Answer 1

Although unrelated to the code, I'll mention that for a CPU to overheat, especially a dual-core CPU, is not usual except with very high ambient temps. I'd suggest removing the heat sink and re-applying thermal paste. Any number of youtube videos can provide step-by-step instructions.

Moving on to the code:

• terminal semicolons aren't needed
• configuration should go at the top
• kill -0 PID is a portable alternative to -e /proc/$pid
• bash builtins let and [[ x -gt y ]] can replace bc for these purposes
• [[ .. ]] is a builtin alternative to [ .. ]
• date +%s can be replaced by builtin printf
• gawk can extract the temperature more flexibly than grep+sed
• your time/run/report pattern can be factored into a function
• the monitoring loop can be simplified by moving sleep to the end
• no real harm in monitoring more aggressively, since the loop is not going to use a lot of CPU
• can save a couple of forks by reading temp directly from /sys, at the cost of CPU-vendor specifity

Putting it all together:

#!/bin/bash
temp_lo=60
temp_hi=95

# reading temps from /sys is CPU-vendor-specific, eliminates need for external sensors program
# temp_label=$( grep -l ^Core /sys/bus/platform/devices/coretemp.*/hwmon/hwmon*/temp*_label  |head -1 )
# temp_source=${temp_label%_label}_input

alias now="printf '%(%s)T\n' -1"

function watch_child {
    childd=$1
    while kill -0 $childd >& /dev/null; do
        temp=$( sensors | gawk -F'[: +°.]+' '/^Core.?1/ {print $3;exit}' )
        # temp=$(( $(<$temp_source) / 1000 ))
        [[ $temp -ge $temp_hi ]] && kill -SIGSTOP $childd
        [[ $temp -le $temp_lo ]] && kill -SIGCONT $childd
        sleep 1
    done
}

function elapsed {
    echo " ******* $1 time: $(date -u -d @$(( ${3:-$(now)}-$2 )) +%T)"
}

function monitor {
    launch_time=${launch_time:-$(now)}
    start_time=$(now)
    echo "********* $1"
    shift
    "$@" &
    watch_child $!
    elapsed Processing $start_time
    elapsed Running $launch_time
}

monitor "The step to perform." my_long_running_command arg1 arg2 
monitor "The next step to perform." another_long_running_command arg1 arg2