# Locating matching files with input folder and file prefix

This is my first more-than-1-line script. It takes an input folder and a file prefix and gets all the matching files. For the first of the files, the script grabs the first line and appends an extra label string and puts that into $data. For all files (including the first), it takes the second line, adds the timestamp found from the filename, and appends a that as new line to $data. When it's done, it writes the output.

The problem is it's pretty slow. 1 directory might have 10,000 files. In that case, the script takes about 3-4 minutes to complete. When doing this over 10 such directories, it begins to get quite annoying.

So, I'm hoping someone here could help speed things up, preferably with explanations so I also get better.

for cs in {47..52}; do
csdirnm="CASE$cs"; tsdirnm="eta_at_x2"; label='flow-time'; dir="$csdirnm/$tsdirnm/"; files=$(ls -tr $dir); # -tr sort on time created, reversed (newest last). i=0; for f in$files ; do
fn=$dir$f;
if [[ $i = 0 ]]; then data="$(sed -n 1p $fn)$label";
((i++));
fi
d=$(sed -n 2p$fn);
t=$(echo$fn | perl -pe 's|.*?(\d+\.\d+)|\1|');
data+="\n$d$t";
done;
echo -e "$data">"$csdirnm/${csdirnm}_${tsdirnm}_20T.time-series";
done;

• Are you running this on GNU/linux? osx? Or are you aiming for the code to be portable? – 200_success Feb 13 '14 at 19:24
• It's running in linux and I would expect it to also be usable on osx. For my part, I don't care about Windows, though a portable solution might be helpful to others using the same unhelpful software (that generates these files), as that runs on Windows also. But concretely, this question is about making it run faster in linux. – AdamAL Feb 13 '14 at 19:32

The problem is almost certainly this line here:

t=$(echo$fn | perl -pe 's|.*?(\d+\.\d+)|\1|');


If you are invoking the Perl interpreter for each line, you will struggle.

A close second is that, for each file, you invoke 2 subshells, and two other program (sed and echo).

My recommendation is for you to actually rewrite the whole thing in Perl...

....

but, you may find it faster to use sed

t=$(echo$fn | sed -e '$s@.*[^0-9]$$[0-9]\+\.[0-9]\+$$.*@\1@g')  The above expression takes advantage of the fact that there has to be some non-digit character before the first date digit. But, as I say, it is my suggestion that you do the whole thing in Perl, and avoid having to do all the execs. EDIT: In Perl #!/usr/bin/perl -w # #for cs in {47..52}; do # csdirnm="CASE$cs";
#  tsdirnm="eta_at_x2";
#  label='flow-time';
#  dir="$csdirnm/$tsdirnm/";
#  files=$(ls -tr$dir); # -tr sort on time created, reversed (newest last).
#  i=0;
#  for f in $files ; do # fn=$dir$f; # if [[$i = 0 ]]; then
#      data="$(sed -n 1p$fn) $label"; # ((i++)); # fi # d=$(sed -n 2p $fn); # t=$(echo $fn | perl -pe 's|.*?(\d+\.\d+)|\1|'); # data+="\n$d $t"; # done; # echo -e "$data">"$csdirnm/${csdirnm}_${tsdirnm}_20T.time-series"; #done; use strict; foreach my$dirnum (47 .. 52) {
my $csdirnm = "CASE$dirnum";
print "Processing Dir $csdirnm\n"; my$tsdirnm = "eta_at_x2";
# Use ls to get the file ordering right.
my $subdir = "$csdirnm/$tsdirnm"; next unless -d$subdir; #Skiip non-existant dirs
open DIRLIST, "-|", "ls -tr $subdir" or die "Unable to list files in$subdir";
my $outfile = "${csdirnm}/${csdirnm}_${tsdirnm}_20T.time-series";
open REPORTFILE, ">", $outfile or die "Unable to write to report file$outfile";

my $file; my$cnt = 0;
while ($file = <DIRLIST>) { chomp$file;
print "Processing $file\n"; my$datafile = "$subdir/$file";
open DATA, "<", $datafile or die "Unable to read file$datafile";
my $line1 = <DATA>; my$line2 = <DATA>;
close DATA;
print REPORTFILE $line1 unless$cnt;
print REPORTFILE $line2;$cnt++;
}

close DIRLIST;
close REPORTFILE;
}

• Re. perl/sed. on my searches to asseble the script, I came across this comparison of perl/sed performance, which indicates that the difference between the two is negligible. Is what you are referring to the 'initilization' if you will? His comparison is using a single call. This is also more in line with my hopes for improving this script. I.e. clever piping, cat'ing and whatnot (which I did try at first, but couldn't get to work with my limited XP). – AdamAL Feb 13 '14 at 17:22
• Re. rewriting completely in perl, I don't think that time saved would weigh up for starting over in yet a new language, that I have little to no knowledge of. – AdamAL Feb 13 '14 at 17:23
• Why don't you try it... I have edited the answer.... – rolfl Feb 13 '14 at 17:45
• Thanks a lot! This is about an order of magnitude (10x) faster. – AdamAL Feb 14 '14 at 7:48
• You forgot the match time-step-in-file-name part. I managed to work it out myself. Was a very good starting point to have your draft to see how thing behave. I'll edit your answer with my modifications. – AdamAL Feb 14 '14 at 8:53