I have a short bash script that processes gigs and gigs of data. I am looking for any improvements to make it faster. This is my very first bash script so please be gentle. I am really only concerned about the while loop. The rest of it is fine I think. It's the while loop where the real work is done and could use the most enhancement.


# This script will clean a WPA wordlist
# It will read every line of the given file
#       Remove all whitespace except for newlines
#       Delete the line if it is less than 8 chars or greater than 63
# It will then exit with time of execution

IFS=$'\n'       # make newlines the only separator

startTime=$(date)       # start time of execution
fileToClean=$1          # this is the file we will be sanatizing, 1st cmd line arg
deletedlines=0          # number of lines that did not meet WPA PSK critera
validlines=0            # number of lines that were valid PSKs  and added to file

if [ -z $fileToClean ]; then #No file specified
        echo ""
        echo "No file specified!"
        echo ""
        exit -1

if [ ! -f $fileToClean ]; then #File does not exist
        echo ""
        echo "File not found!"
        echo ""
        exit  -1

#By this point I am assuming I entered a valid file and will begin cleaning
echo ""
echo 'Cleaning word list: ' $1
echo "Start Time: " $startTime
echo ""

while read line; do  #read every line in file and save to var line

        line="${line##*( )}"        # trim leading whitespace
        line="${line%%*( )}"        # trim trailing whitespace

        if [ ${#line} -ge 8 ] && [ ${#line} -le 63 ]; then     # if trimmedline length >= 8 && <= 63
                echo  $line >> $outputfile


done <$fileToClean

utime="$( TIMEFORMAT='%lU';time ( ls ) 2>&1 1>/dev/null )"  # this stores the executio time in the var utime

echo "Processing completed, it took" $utime "and $deletedlines were deleted."
echo $validlines "were added to the output file "$outputfile" as they were valid PSKs"
echo ""

migrated from programmers.stackexchange.com Mar 20 '15 at 23:19

This question came from our site for professionals, academics, and students working within the systems development life cycle.

  • \$\begingroup\$ The method you are using in your while loop you involves launching a process every time you want to trim lines. I guess this is going to be a bottleneck. You could try trimming the lines using bash-only constructs and see if this improves your throughput. See here for ideas (see the "trim" function suggestion) stackoverflow.com/questions/369758/… \$\endgroup\$ – Brandin Mar 20 '15 at 23:09
  • \$\begingroup\$ Is there a reason to only use bash for text processing? I wrote a roughly equivalent loop in Python and ran it on a 100mb file of base64 data ("roughly equivalent" because I will write to the output file if the line starts with 'j') and it takes a total .5 seconds. The bash version is still running after 5 minutes. \$\endgroup\$ – Landon Mar 20 '15 at 23:18
  • \$\begingroup\$ I would like to stick to bash as I do not know any Python and feel using the right constructs I will be able to make a reasonable fast bash script for this task. I see how fast your method was +1 but I have a desire to stick to bash as this is also some what of a learning session for me. Thanks for the idea landon! @Brandin I will take your idea under consideration and see what you mean, I will look into trying a different method and will report back. \$\endgroup\$ – Dylan Mar 21 '15 at 2:25
  • \$\begingroup\$ The script is done running, what took hours to do a portion of the file took minutes using bash commands instead of launching external commands. Can you create an answer with what you recommended. \$\endgroup\$ – Dylan Mar 21 '15 at 4:19
  • \$\begingroup\$ I am sorry the file I used as test input was not 4gb it was 139921497 bytes( .1399gb ). My mistake, but I was correct when I stated it took hours to run on the original script and only completed 4 millions lines and this time it completed all 14 million lines in just minutes. \$\endgroup\$ – Dylan Mar 21 '15 at 5:03

The if statement can be reduced to just:

[[ ${#line} >= 8 && ${#line} <= 63 && $((++validLines)) > 0 ]] 
    && echo $line >> $outputfile || ((deletedlines++))

We use bash's [[ ... ]] shell keyword to collate our conditions together. Helpfully, [[ ... ]] supports the use of >= and <=.

[[ <something-to-test> ]] && <one-command-if-test-is-true> || <one-command-if-test-is-false> is quite a useful (IMHO!) construct in bash for implementing a if-then-else logic. The only thing to note is that you can only specify one command each for both branches.

As such, we can 'inline' our increment of validLines and do an always true comparison to simplify our if branching, namely to either (&&) output the line to the output file, or (||) increment our deletedLines. Otherwise, it gets slightly longer due to the required use of { ... } compound command:

[[ ${#line} >= 8 && ${#line} <= 63 ]] 
    && { ((validLines++)); echo $line >> $outputfile; } || ((deletedlines++))

Now onto the other parts...

  • fileToClean should be set as fileToClean="$1" in case the filename has spaces inside.
  • Multi-line echo can be done as such:

    [ ! -f "$fileToClean" ] && cat<<M && exit -1
    Oh noes,
    We have a multi-line error message
    Telling user that the file is not a regular file.
  • Your way of checking the elapsed time should be done outside your script, using the time command. time can generate more (and probably more accurate) statistics about the 'performance' of your script and is readily available, hence the suggestion as opposed to manually calculating the value(s) yourself. :)

Now do you really need to do this in bash? Here's an awk one-liner (split into three for readability):

awk '{sub(/^[ ]+/,"");sub(/[ ]+$/,"")};
    length()>=8&&length()<=63{v++;print > "'"$outputfile"'" };
    END{print "Valid lines:",v"\nInvalid lines:",NR-v}' "$fileToClean"
  1. Trim line.
  2. If length matches, count the line and print to "$outputfile".
  3. At the end, print the summary for valid/invalid lines.
  • \$\begingroup\$ For me this is some dense coding, I am gonna have to work through all of this. For example I have never used the shell keyword, I have also never learned awk though it seems like I should.... When I have some time tomorrow I will work through all this. Though it seems elegant and effective. +1 and probably accepted \$\endgroup\$ – Dylan Mar 27 '15 at 7:13
  • 1
    \$\begingroup\$ Think of [[ as a super [... ;) \$\endgroup\$ – h.j.k. Mar 27 '15 at 7:20
  • 1
    \$\begingroup\$ Ok so it is a keyword not a command, that answered what I was looking for. I am gonna read over both of those for sure. I am just trying to suck up everything I can right now. Learning bash, python and mongo all day so I am spent. But I will get back on it tomorrow and like I said I am sure I will be accepting your answer! Thank you that is great! \$\endgroup\$ – Dylan Mar 27 '15 at 7:46
  • 1
    \$\begingroup\$ I don't have much experience dealing with non-UTF-8 characters, but maybe this will give you a start? stackoverflow.com/questions/16760493/… \$\endgroup\$ – h.j.k. Mar 28 '15 at 4:37
  • 1
    \$\begingroup\$ It is amazing how powerful awk really is. It is basically C without having to worry about a lot of the details with the language.... Thanks again \$\endgroup\$ – Dylan Mar 28 '15 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.