2
\$\begingroup\$

I have huge number of text files, each several hundred MB in size. Unfortunately, they are not all fully standardized in any one format. Plus there is a lot of legacy in here, and a lot of junk and garbled text. I wish to check all of these files to find rows with a valid email ID, and if it exists then print it to a file named the first-char of the email ID. Hence, multiple text files get parsed and organized into files named a-z and 0-9. In case the email address starts with a special character, then it will get written into a file called "_" underscore. The script also trims the rows to remove whitespaces; and replaces single and double quotes (this is an application requirement)

My script works fine. There are no errors or bugs in it. But it is incredibly slow. My question: is there a more efficient way to achieve this? Parsing 30 GB logs takes me about 12 hrs - way too much! Will grep/cut/sed/another be any faster?

Sample txt File

[email protected],address
#[email protected];address
[email protected];address µÖ
[email protected];username;address
[email protected];username
  [email protected],username;address   [spaces at the start of the row]
 [email protected]|username|address   [tabs at the start of the row]

My Code:

awk -F'[,|;: \t]+' '{
    gsub(/^[ \t]+|[ \t]+$/, "")
    if (NF>1 && $1 ~ /^[[:alnum:]_.+-]+@[[:alnum:]_.-]+\.[[:alnum:]]+$/)
    {
        gsub(/"/, "DQUOTES")
        gsub("\047", "SQUOTES")
        r=gensub("[,|;: \t]+",":",1,$0)
        a=tolower(substr(r,1,1))
        if (a ~ /^[[:alnum:]]/)
            print r > a
        else
            print r > "_"
    }
    else
        print $0 > "ErrorFile"
}' *.txt
\$\endgroup\$
1
  • \$\begingroup\$ Is it necessary to keep the entire line or just the email address? Also, does case matter to you? That is, would it hurt if all characters were converted to lowercase? \$\endgroup\$
    – Edward
    Jul 6, 2020 at 15:56

2 Answers 2

1
\$\begingroup\$

12 hrs - way too much!

It's not clearly too much. You're processing a Billion rows at a rate of 25,000 per second. Spending a mere 40 µsec per row is not too shabby.

Even if we shaved it to 20 µsec, you likely would still grumble that six hours is too much.


Switching to Rust or Haskell would be faster, but that's a pretty big coding change.

Switching to a compiler / JIT language in the JS node family might be more attractive. I hear the raku project devotes some effort to ensuring they bench well on this type of task.

Let us know how any revised benchmarks go.


Kudos on using ^ caret and $ dollar anchors where you can. It prevents the regex engine from getting distracted and doing useless work that's immediately discarded.


If you can patiently wait for split -l ... to read 30 GiB, you can break this into sub-problems that can be sent to various cloud hosts in parallel.

Or write a short program that seeks to offset and scans to next newline, so that even the splitting happens in parallel.


To the extent that you're I/O limited, processing compressed input and compressed output files can be helpful, as there's fewer disk blocks to move around.

Instead of 27 output files, consider writing to a single file ordered by /usr/bin/sort. Scan the result once and write a brief ToC table of contents, which says "seek to zero for A", "seek to NNN for B", and so on. Notice that utf-8 processing can be skipped with env LC_ALL=C sort ..., for a significant speed gain.

            print r > "_"

It's not obvious to me if we're sticking with chapter 3 userspace calls, or if that incurs the overhead of a write(2). Possibly buffering up some output lines could reduce such overhead.


        r = gensub("[,|;: \t]+", ":", 1, $0)

We ask the regex engine to re-compile same old expression a Billion times. So this has higher overhead than / ... / syntax, which in perl and awk will cache the compiled result.

Also, didn't we previously use -F so the input field separator has already accomplished that work? We have all the fields, just stitch them together with : colons in between.


there is a lot of legacy in here, and a lot of junk and garbled text.

It's unclear what fraction of rows are just unusable garble. If it is "a lot", then preprocess with

    ... | grep @ | awk ...

so the awk script has fewer rows to discard. Nothing will filter faster than grep.

\$\endgroup\$
0
\$\begingroup\$

Don't know if this is affecting your performance, but one thing to check is that you are using an up-to-date version of awk. In particular, the version of awk that ships on MacOS is very slow. You may get dramatic difference by installing the latest GNU gawk. I've measured order of magnitude performance improvements by upgrading. (On MacOS, Homebrew and MacPorts are two popular package managers that can install the latest version.)

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.