Bash grep taking 3 days to finish. Any suggestions to enhance it?

Question

I have a script like this that I would like to seek some suggestions on enhancing it.

cd /home/output/
cat R*op.txt > R.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R.total.op.txt > temp && mv temp R.total.op.txt

cat S*op.txt > S.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' S.total.op.txt > temp && mv temp S.total.op.txt

cat R.total.op.txt S.total.op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' > genes.txt
rm *total.op.txt
head genes.txt

cd /home/output/
for j in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
***** for i in `cat genes.txt`; do cat $j'.op.txt' | grep -w $i >> $j'_'$i'_gene.txt';done
done
ls -m1 *gene.txt | wc -l
find . -size 0 -delete
ls -m1 *gene.txt | wc -l
rm genes.txt

cd /home/output/
for i in `ls *gene.txt`
do
paste <(awk '{print $4"\t"$8"\t"$9"\t"$13}' $i | awk '!x[$1]++' | awk '{print $1}') <(awk '{print $4"\t"$8"\t"$9}' $i | awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | awk '{ sum += $3 } END { if (NR > 0) print sum / NR }') <(awk '{print $4"\t"$8"\t"$9}' $i| awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | wc -l) <(awk '{print $4"\t"$8"\t"$9"\t"$13}' $i | awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | grep -v ":::" | wc -l) > $i'_stats.txt'
done
rm *gene.txt

cd /home/output/
for j in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
cat $j*stats.txt > $j'.final.txt'
done
rm *stats.txt

cd /home/output/
for i in `ls *final.txt`
do
sed "1iGene_Name\tMean1\tCalculated\tbases" $i > temp && mv temp $i
done
head *final.txt

The second for loop (marked with asterisks) that has cat genes.txt is the grep loop that is taking 3 days to finish. Can someone please advice any enhancements to the command and if this entire script can be made into a single command? Thanks in advance. I posted in stackoverflow and was asked to send it here. :)

Answer 1

Without your data, it's difficult to test if the changes help at all. I can only assume that the constant reading and writing to files on disk is slowing down your program. If your data is simply big enough to require 3 days to run in a bash script, then you may have to switch to another programming language with more complex data structures that can do your processing faster.

The first thing I notice is that you have cat and temporary files everywhere. The programs you are piping cat output to can take file name arguments. Plus, writing to disk is one of the slowest things you can do to move data around. For example, all this

cat R*op.txt > R.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R.total.op.txt > temp && mv temp R.total.op.txt

cat S*op.txt > S.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' S.total.op.txt > temp && mv temp S.total.op.txt

cat R.total.op.txt S.total.op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' > genes.txt
rm *total.op.txt

can be replaced with the following with only one file write

awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R*op.txt S*op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' > genes.txt

Now, for the loop you marked

for j in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
    for i in `cat genes.txt`
    do
        cat $j'.op.txt' | grep -w $i >> $j'_'$i'_gene.txt'
    done
done

Try changing this to (with some better names)

for file in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
    while read -r line
    do
         grep -w $line $file'.op.txt' >> $file'_'$line'_gene.txt'
    done < genes.txt
done

This will read the file line by line instead of all at once with cat genes.txt. An important note is that this will give different results than your script if there are multiple words per line in genes.txt. To read word-by-word, use

for file in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
    while read -r line
    do
        for word in $line
        do
            grep -w $word $file'.op.txt' >> $file'_'$word'_gene.txt'
        done
    done < genes.txt
done

If genes.txt is much larger than the R1_with-genename, etc., files, then reversing the order of the loops could help by only reading the larger file once.

while read -r line
do
    for file in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
    do
         grep -w $line $file'.op.txt' >> $file'_'$line'_gene.txt'
    done
done < genes.txt

Using >> appends to the end of the file if the file exists. Is this what you want? If running the script should overwrite previous results, use a single >.

Also, since you delete the genes.txt file right afterward, you don't even have to create it.

awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R*op.txt S*op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' | while read -r line
do
    for file in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
    do
         grep -w $line $file'.op.txt' >> $file'_'$line'_gene.txt'
    done
done