I need to run CPU-intensive tasks on a very old machine with overheating issues. The script below will monitor temperature and pause the jobs when it gets too high, continuing them when it's back to normal.
The actual commands run are, of course, not included, since they are irrelevant to the question.
I am looking for hidden traps I may have set in my code (listed at the bottom), and for other things I have done incorrectly. Aside from special characters in the commands and arguments that are run, which are hand created so I can control that risk, what traps or "got-ya's" have I unknowingly set into the code? What ways are there for making this more error-proof, or better in other ways?
For the timing function I know I could have used the
time { command ...; command ...; }
construct, but I was more interested in the time spent by the machine (and previously, by me in the chair) than in the CPU time involved.
The Script:
The code comments should explain what it does, as well as why I did some of it the way I did.
#!/bin/bash
# Build my time reporting function
function report {
# Get the current time, do the math, report the results.
end_time=$(date +%s);
# The time used for the last run process
proc_time=$(echo "$end_time"-"$start_time" | bc);
echo " ******* Processing time: $(date -u -d @${proc_time} +%T)";
# The cummulative time for all processes so far
run_time=$(echo "$end_time"-"$launch_time" | bc);
echo " ******* Running time: $(date -u -d @${run_time} +%T)";
}
# The high and low temperatures to monitor for. Processing is paused
# once the high temp is reached, and will not resume again until the
# low temp is reached.
# My system recovers to 60°C reasonably quick (idle is around 45°C)
temp_lo=60;
# My system dies at about 115°C - since 100°C is normal, suggests my
# sensors are not accurate, but I work with what I have.
# 20°C margin allows for delay in the detection of the high temp, and
# delay in the process pausing, while still keeping temp under danger
# zone. Also allows for when Core0 is rising faster than Core1. They
# seem to take turns being the leader, but seldom more than 5-10°C
# difference.
temp_hi=95;
# The routine to read the CPU temp with lm sensors. Could be coded
# inline in the watch_child function, but that means placing it in
# three places, and if the grep/sed needs adjusting, then I have to
# remember to change _all_ three, and not make any typos. This cuts
# my chance of errors to a third.
function get_temp {
# the grep and/or sed may need changing for other sensor output
# on different systems
sensors | grep 'Core1' | sed -e 's/.*: \+\([+-][0-9.]\+\)°C.*$/0\1/'
}
# Routine to monitor the CPU temp, pausing the processing as needed
# to remain in the 'safe' range for processor temperature.
function watch_child {
# argument should be the PID of the backgrounded process
childd=$1;
# pre-load the CPU temp
temp=$(get_temp);
# As long as the backgrounded process is still running
while [ -e /proc/$childd ]; do
# Monitor the process, for still running, and the temp, still
# safe
while [ -e /proc/$childd ] && [ $(echo "$temp < $temp_hi" | bc) = 1 ]; do
# wait a spell
sleep 5;
# re-load the temp for a re-check
temp=$(get_temp);
done
# If the process is still running, then it was over-temp that
# caused the while loop to end
if [ -e /proc/$childd ]; then
# Tell the process to take a break
kill -SIGSTOP "$childd";
fi
# Drops through here if the process has ended, otherwise,
# monitor the temp for a restart
while [ -e /proc/$childd ] && [ $(echo "$temp > $temp_lo" | bc) = 1 ]; do
# wait a spell
sleep 5;
# re-load the temp for a re-check
temp=$(get_temp);
done
# Drop through here if the process has ended.
if [ -e /proc/$childd ]; then
# Otherwise, tell the process that the break is over.
kill -SIGCONT "$childd";
fi
done
# Only get this far once the process has ended.
# In the rare case of the process never waking up, the outer while
# loop will run infinitely!
# Human monitoring still required!
}
# Start the timer for cumulative run time reports
launch_time=$(date +%s);
echo "********* The step to perform.";
# Start the timer for this process
start_time=$(date +%s);
# Launch the dangerous process in the background
my_long_running_command arg1 arg2 &
# Capture its PID
child=$!;
# Block, with temp throttling, until this process is done
watch_child $child;
report;
echo "********* The next step to perform.";
# Start the timer for the next process
start_time=$(date +%s);
# Launch the dangerous process in the background
another_long_running_command arg1 arg2 &
# Capture its PID
child=$!;
# Block, with temp throttling, until this process is done
watch_child $child;
report;
