ECG Bash selection tool

Question

I made the following bash script for extracting a group of ECG signals from ECG files. I would like to know if there is any mistakes and/or weaknesses. I have experienced difficulties in integrating bash parameters to it as a function because of AWK part.

I think it would be better not to use so much different separate tools because of such problems, but not sure how to replace, for instance, the AWK part by something more stable together with bash.

Each ECG file contains two columns where the first column is the original signal and the second column is the improved ECG signal.

The database is AAMI MIT-BIH Arrhythmia. The script must be stable and must be valid, so I have not used wildcard characters there. The users give IDs which they want. They give also which ECG signal they want (1 or 2).

Now, the type of ECG signal has to be manually corrected because I cannot integrate $ecg in awk one-liner.

Logic of the script:

1. Get a list of wanted ECG columns into ECGs; there is a repetition of the ID 118 because repetition should be allowed and duplicate IDs should not removed
2. Greate and/or empty temporary files; keep iteration individual ECG in /tmp/test.csv and the combination result in result.csv
3. Loop through ECGs to have them in result.csv
4. Add a header to the beginning of the file by ids

getEcgs.bash

#!/bin/bash
ids=(101 118 201 103 118)
dir="/home/masi/Documents/CSV/"
#Ecgs=()
index=0
ecg=2 # ecg=1 ecg; ecg=2 improved ecg # change AWK line $2/$1 to corresponding number manually for change; buggy AWK with bash params

#printf '%s\n' "${#ids[@]}"
#printf '%s\n' "${ids[0]}"
#printf '%s\n' "${ids[1]}"

for id in "${ids[@]}";
do  
    input=$(echo "${dir}P${id}C1.csv")
    # take second column of the file here
    file=$(awk -F "\"*,\"*" '{print $2}' $input) # http://stackoverflow.com/a/19602188/54964 # http://stackoverflow.com/a/19075707/54964

#   printf '%s\n' "${id}"
#   printf '%s\n' "$index"

    Ecgs[${index}]="${file}"

    index=$index+1
done

#declare -A "${Ecgs[@]}"
#printf '%s\n' "${Ecgs[@]}" # http://stackoverflow.com/a/15692004/54964

#printf '%s\n' "${#Ecgs[@]}"

filenameTmp=/tmp/test.csv
filenameTarget=/tmp/result.csv
:> "$filenameTmp" # https://unix.stackexchange.com/a/320142/16920
:> "$filenameTarget"

# Put array items columnwise into .csv file
let N="${#Ecgs[@]}"-1 # https://unix.stackexchange.com/a/149832/16920
for index in `seq 0 "${N}"`;
do
    printf '%s\n' "${Ecgs[${index}]}" > "${filenameTmp}"

    if [[ "${index}" -eq 0 ]]; then 
        cat "${filenameTmp}" > "${filenameTarget}"
    fi 

#   cat "${filenameTmp}"

#   paste <(cat /tmp/result.csv) <(cat /tmp/test.csv) > /tmp/result.csv
    if [[ "${index}" > 0 ]]; then 
        paste -d "," "${filenameTarget}" "${filenameTmp}" | column -s $'\t' -tn > "${filenameTarget}" # https://unix.stackexchange.com/a/16465/16920
    fi
done

header=$(printf ",%s" ${ids[@]}) # http://stackoverflow.com/a/2317171/54964
header=${header:1} # to remove the first comma caused by printf
sed -i "1s/^/${header}\n/" "${filenameTarget}"

Input data examples /home/masi/Documents/CSV/P100C1.csv, P101C1.csv, P118C1.csv and P201C1.csv:

masi@masi:~/Documents/CSV$ head -n +5 P101C1.csv 
-0.56,1.61
-0.575,0.67
-0.56,0.695
-0.545,0.38
-0.52,0.43

masi@masi:~/Documents/CSV$ head -n +5 P100C1.csv 
-0.295,-0.465
-0.295,-0.44
-0.295,-0.435
-0.31,-0.425
-0.315,-0.41

masi@masi:~/Documents/CSV$ head -n +5 P118C1.csv 
-0.69,-1.84
-0.67,-0.71
-0.67,-0.49
-0.69,-0.26
-0.74,0.07

masi@masi:~/Documents/CSV$ head -n +5 P201C1.csv 
-0.21,-0.245
-0.205,-0.22
-0.225,-0.2
-0.22,-0.2
-0.21,-0.195

Temporary file content in /tmp/test.csv for a single iteration

-1.84
-0.71
-0.49
-0.26

Output /tmp/result.csv with headers where you see repetition of ID 118 occurs as expected:

101,118,201,103,118
1.61,-1.84,-0.245,-0.405,-1.84
0.67,-0.71,-0.22,-0.32,-0.71
0.695,-0.49,-0.2,-0.32,-0.49

Discussions about some parts of the script:

• Using AWK with Bash parameters here; AWK is not critical for the script but causes difficulties in handling Bash parameters; so it can be replaced such that the script could be made a function.

  # https://unix.stackexchange.com/a/320857/16920
  # not working in this application
  #paste -d"," ${input[@]} | awk -F, -v OFS=, '{print $2, $4, $6}' > /tmp/testShort.csv
  

OS: Debian 8.5

Bash: 4.30

Answer 1

Shell scripts that do complex line-oriented text processing using Awk and other tools are usually better done using Awk alone. Not only would the script be more efficient, it would be more coherent, and have fewer quoting issues. Consider the following script, which I'll call ecg:

#!/usr/bin/gawk -f

# https://www.gnu.org/software/gawk/manual/html_node/Join-Function.html
@include "join.awk"

BEGIN {
    FS = "\"*,\"*";
    last_row = 0;
}

BEGINFILE {
    rows[0][ARGIND] = gensub(".*P([0-9]*)C.*", "\\1", "g", FILENAME);
}

{
    rows[FNR][ARGIND] = $col;
    if (FNR > last_row) { last_row = FNR; }
}

END {
    for (r = 0; r <= last_row; r++) {
        print join(rows[r], 1, ARGC - 1, ",");
    }
}

Observe what happens when you run it:

$ ./ecg -v col=2 P{101,118,201,118}C1.csv
101,118,201,118
1.61,-1.84,-0.245,-1.84
0.67,-0.71,-0.22,-0.71
0.695,-0.49,-0.2,-0.49
0.38,-0.26,-0.2,-0.26
0.43,0.07,-0.195,0.07

Note that $col extracts the column specified by the parameter col.

Since you are using GNU/Linux, I have taken advantage of some features specific to GNU Awk in the script above:

• Multidimensional arrays. Traditional Awk only has one-dimensional arrays which can be indexed using tuples to simulate extra dimensions.
• The BEGINFILE special pattern and the ARGIND special variable.
• The gensub() function to extract the ID from the filename.
• The join() function.