0
\$\begingroup\$

I have a script like this that I would like to seek some suggestions on enhancing it.

cd /home/output/
cat R*op.txt > R.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R.total.op.txt > temp && mv temp R.total.op.txt

cat S*op.txt > S.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' S.total.op.txt > temp && mv temp S.total.op.txt

cat R.total.op.txt S.total.op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' > genes.txt
rm *total.op.txt
head genes.txt

cd /home/output/
for j in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
***** for i in `cat genes.txt`; do cat $j'.op.txt' | grep -w $i >> $j'_'$i'_gene.txt';done
done
ls -m1 *gene.txt | wc -l
find . -size 0 -delete
ls -m1 *gene.txt | wc -l
rm genes.txt

cd /home/output/
for i in `ls *gene.txt`
do
paste <(awk '{print $4"\t"$8"\t"$9"\t"$13}' $i | awk '!x[$1]++' | awk '{print $1}') <(awk '{print $4"\t"$8"\t"$9}' $i | awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | awk '{ sum += $3 } END { if (NR > 0) print sum / NR }') <(awk '{print $4"\t"$8"\t"$9}' $i| awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | wc -l) <(awk '{print $4"\t"$8"\t"$9"\t"$13}' $i | awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | grep -v ":::" | wc -l) > $i'_stats.txt'
done
rm *gene.txt

cd /home/output/
for j in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
cat $j*stats.txt > $j'.final.txt'
done
rm *stats.txt

cd /home/output/
for i in `ls *final.txt`
do
sed "1iGene_Name\tMean1\tCalculated\tbases" $i > temp && mv temp $i
done
head *final.txt

The second for loop (marked with asterisks) that has cat genes.txt is the grep loop that is taking 3 days to finish. Can someone please advice any enhancements to the command and if this entire script can be made into a single command? Thanks in advance. I posted in stackoverflow and was asked to send it here. :)

\$\endgroup\$
  • 1
    \$\begingroup\$ It would be best to edit the question and title to describe what the script does, you're likely to get more positive feedback. :) \$\endgroup\$ – 410_Gone Jun 30 '17 at 16:12
  • 2
    \$\begingroup\$ What is the purpose of this script? Please make that the title. That's how we do things around here. Please also read How to Ask. \$\endgroup\$ – Stop ongoing harm to Monica Jun 30 '17 at 17:03
  • 1
    \$\begingroup\$ Hi - The script reads a multi column file that has gene names in it. And then it follows a filtering protocol using awk leading to a file called op.txt. Using these op.txt files, I consider the unique genes. And for every gene, I want to know if it exists in the file that it is processing and if it does, I want it to print it into a file by putting the genename into the filename. This particular grep step is taking days to finish. Was wondering if there is any enhancement to this. Stackoverflow guys helped with awk scripts. But I gotta test it. Will let you know shortly. \$\endgroup\$ – j smith Jun 30 '17 at 18:00
  • 1
    \$\begingroup\$ How big is the input data? \$\endgroup\$ – chicks Jun 30 '17 at 18:27
3
\$\begingroup\$

Without your data, it's difficult to test if the changes help at all. I can only assume that the constant reading and writing to files on disk is slowing down your program. If your data is simply big enough to require 3 days to run in a bash script, then you may have to switch to another programming language with more complex data structures that can do your processing faster.

The first thing I notice is that you have cat and temporary files everywhere. The programs you are piping cat output to can take file name arguments. Plus, writing to disk is one of the slowest things you can do to move data around. For example, all this

cat R*op.txt > R.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R.total.op.txt > temp && mv temp R.total.op.txt

cat S*op.txt > S.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' S.total.op.txt > temp && mv temp S.total.op.txt

cat R.total.op.txt S.total.op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' > genes.txt
rm *total.op.txt

can be replaced with the following with only one file write

awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R*op.txt S*op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' > genes.txt

Now, for the loop you marked

for j in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
    for i in `cat genes.txt`
    do
        cat $j'.op.txt' | grep -w $i >> $j'_'$i'_gene.txt'
    done
done

Try changing this to (with some better names)

for file in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
    while read -r line
    do
         grep -w $line $file'.op.txt' >> $file'_'$line'_gene.txt'
    done < genes.txt
done

This will read the file line by line instead of all at once with cat genes.txt. An important note is that this will give different results than your script if there are multiple words per line in genes.txt. To read word-by-word, use

for file in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
    while read -r line
    do
        for word in $line
        do
            grep -w $word $file'.op.txt' >> $file'_'$word'_gene.txt'
        done
    done < genes.txt
done

If genes.txt is much larger than the R1_with-genename, etc., files, then reversing the order of the loops could help by only reading the larger file once.

while read -r line
do
    for file in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
    do
         grep -w $line $file'.op.txt' >> $file'_'$line'_gene.txt'
    done
done < genes.txt

Using >> appends to the end of the file if the file exists. Is this what you want? If running the script should overwrite previous results, use a single >.

Also, since you delete the genes.txt file right afterward, you don't even have to create it.

awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R*op.txt S*op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' | while read -r line
do
    for file in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
    do
         grep -w $line $file'.op.txt' >> $file'_'$line'_gene.txt'
    done
done
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.