We have a process that outputs to log files in the format /var/log/xxx/YYYY_MM_DD.log
- the file name is chosen programmatically and is not rotated using logrotate or anything like that. We want to automatically tail the latest file and pipe that to slackcat so that we can see the logs in a certain slack channel.
I came up with the following solution that works great, but I'm no bash expert so I am wondering if this could be simplified? The hardest part was avoiding zombie processes when something dies or gets killed, but I'd love to do this with one file instead of 2 also:
watchlogdir.sh
#!/bin/sh
trap "pkill -TERM -g $$; exit" INT TERM EXIT
while true; do
/root/tailtoslack.sh &
PID=$!
echo $PID
echo $$
inotifywait -e create /var/log/xxx/
pkill -TERM -P $PID
kill $PID
done
tailtoslack.sh
while true; do
FILETOWATCH=`ls -t /var/log/xxx/*.log | head -1`
tail $FILETOWATCH -f -n1 | grep -v "DEBUG:\|^$" --color=never --line-buffered | /root/slackcat &> /tmp/slackcat
sleep 31
done
The core of watchlogdir.sh
, besides the interruption handling stuff, is using inotifywait
to watch for a new file created in the log directory, and when that happens, killing tailtoslack.sh
and respawning it so it can find the new file
tailtoslack.sh
just looks for the latest file, and pipes that to slackcat after filtering out DEBUG
lines and empty lines. the while loop here is because some lines in the error logs cause slackcat
to crash, so this way if that happens it'll sleep for a little while and retry.
Without changing the requirements of how the log files are written, can any of this be done better?