1
\$\begingroup\$

I have the code below, which works successfully, and is used to parse, clean log files (very large in size) and output into smaller sized files. Output filename is the first 2 characters of each line. However, if there is a special character in these 2 characters, then it needs to be replaced with a '_'. This will help ensure there is no illegal character in the filename.

This would take about 12-14 mins to process 1 GB worth of logs (on my laptop). Can this be made faster?

For example, would it help to run this parallel? I am aware I could do }' "$FILE" &. However, I tested and that does not help much. Perhaps AWK itself could run in parallel (the equivalent of print $0 >> Fpath & )?

Sample log file

email1@foo.com:datahere2     
email2@foo.com:datahere2
email3@foo.com datahere2
email5@foo.com;dtat'ah'ere2 
wrongemailfoo.com
nonascii@row.com;data.is.junk-Œœ
email3@foo.com:datahere2

Expected Output

# cat em 
email1@foo.com:datahere2     
email2@foo.com:datahere2
email3@foo.com:datahere2
email5@foo.com:dtat'ah'ere2 
email3@foo.com:datahere2

# cat errorfile
wrongemailfoo.com
nonascii@row.com;data.is.junk-Œœ

Code:

#/bin/sh
pushd "_test2" > /dev/null
for FILE in *
do
    awk '
    BEGIN {
        FS=":"
    }
    {
        gsub(/^[ \t"'\'']+|[ \t"'\'']+$/, "")
        $0=gensub("[,|;: \t]+",":",1,$0)
        if (NF>1 && $1 ~ /^[[:alnum:]_.+-]+@[[:alnum:]_.-]+\.[[:alnum:]]+$/ && $0 ~ /^[\x00-\x7F]*$/)
        {
            Fpath=tolower(substr($1,1,2))
            Fpath=gensub("[^[:alnum:]]","_","g",Fpath)
            print $0 >> Fpath
        }
        else
            print $0 >> "errorfile"
    }' "$FILE"
done
popd > /dev/null
\$\endgroup\$
4
  • \$\begingroup\$ To the reviewers: I think this question has been edited into shape good enough that it's within the scope of the site. Please leave a comment if you disagree. \$\endgroup\$
    – Mast
    Jul 3 '20 at 6:07
  • \$\begingroup\$ Mast: i disagree. your edits wont make the code run any faster. in future pls post your answer separately, and not edit the original ques \$\endgroup\$
    – rogerwhite
    Jul 3 '20 at 13:56
  • \$\begingroup\$ @rogerwhite You may find this blog post helpful: aosabook.org/en/posa/… .. \$\endgroup\$
    – aki
    Jul 5 '20 at 11:45
  • \$\begingroup\$ @akki - sorry for the slow revert. all good, but i need help with the code !! \$\endgroup\$
    – rogerwhite
    Jul 25 '20 at 14:17
1
\$\begingroup\$

Perhaps switching the outputfile for each line can be avoided by sorting the inputfiles first or with some changes in your script:

# Store the 2-letter filenames
outfile[Fpath];
# Store the highest index for given outputfile
i[Fpath]++;
# Store current line in output array for that file
a[Fpath][i[Fpath]]=$0
# And in the END block print for array per output file
      for (out in outfile) {
         for (j=1;j<=i[out]; j++) {
            print a[out][j] >> out;
         }
      }

This results in

awk '
    BEGIN {
        FS=":"
    }
    {
        gsub(/^[ \t"'\'']+|[ \t"'\'']+$/, "")
        $0=gensub("[,|;: \t]+",":",1,$0)
        if (NF>1 && $1 ~ /^[[:alnum:]_.+-]+@[[:alnum:]_.-]+\.[[:alnum:]]+$/ && $0 ~ /^[\x00-\x7F]*$/)
        {
            Fpath=tolower(substr($1,1,2))
            Fpath=gensub("[^[:alnum:]]","_","g",Fpath);
            outfile[Fpath];
            i[Fpath]++;
            a[Fpath][i[Fpath]]=$0
        }
        else
            print $0 >> "errorfile"
    }
   END {
      for (out in outfile) {
         for (j=1;j<=i[out]; j++) {
            print a[out][j] >> out;
         }
      }
   } ' "$FILE"
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.