5
\$\begingroup\$

I'm aware I am probably reinventing the wheel somewhat here, but I am trying to teach myself simple bash coding by completing simple tasks such as parsing files.

To that end I am looking to learn what elephants in the room I could be missing or if there are better ways to use the core functionality of bash without installing any additional tools.

This simple code returns a list of unique IP addresses that have hit the index of my site along with a count of the hits.

a="access.log"; for b in $(cat $a | awk '{print $1}' | sort | uniq);do echo $b;grep $a -e "GET / HTTP" | grep -c $b;done;

Assumptions: access.log is in the current directory and is in the regular format

Any advice or suggestions for improvement are greatly appreciated

\$\endgroup\$
5
\$\begingroup\$

Well, your code is hardly a bash solution, is it? You use sort, awk, grep, and echo....

Additionally, your code is dumped on a single line, and it makes it hard to read. Why not put it in a script, and have separate commands on separate lines.... like:

#!/bin/bash

a="access.log"
for b in $(cat $a | awk '{print $1}' | sort | uniq); do
  echo $b;
  grep $a -e "GET / HTTP" | grep -c $b;
done;

Those variable names.... ouch. a and b make it hard to separate from the -c and -e too.... and they mean nothing otherwise. Why not use meaningful names like ip and log?

Then, when I ran the code, I got a lot of funny results.... like:

54.69.125.145
1
61.240.144.65
0
64.14.99.254
0
66.196.235.78
0
66.249.64.188
0
74.208.152.232
0

Why are there 0 counts.... oh, that's because those are IP's that are not accessing the home page, but are accessing other pages... they appear as $b but don't actually "GET" /.

I would consider making it more a study of bash and use the native bash structures to get things right.... no grep, awk, etc.

#!/bin/bash

# use first commandline argument if supplied
log="access.log"
if [ $1 ] ; then
    log="$1"
fi

# set a variable to match in a regular expression
match="GET / HTTP"

# create a named array.
declare -A counts

# read the file line-by-line
while IFS='' read -r line || [[ -n "$line" ]]; do

  # find lines that access GET / HTTP
  if [[ $line =~ $match ]] ; then

    # get just the IP of the client
    ip=${line%% *}
    # get the previous count, default to 0
    prev=${counts[$ip]:-0}
    # increment the count for this IP
    counts[$ip]=$(($prev + 1))
  fi
done < "$log"

for ip in "${!counts[@]}" ; do
    echo "IP $ip visited ${counts[$ip]} times"
done

EDIT: About the ${line%% *} variable substitution. The possibilities when doing variables in bash are remarkably powerful. I recommend looking at the document Parameter Substitution for details, and the man page for is good as well (but does not have the examples). The %% token indicates that there should be a pattern search backward from the end of $line for a space followed by any characters (the * - this is a "glob" expression, not a regex). This pattern essentially looks for the first space, and removes it and any charaters after it. The man page document says:

${parameter%word}
${parameter%%word}

    The word is expanded to produce a pattern just as in filename expansion.
    If the pattern matches a trailing portion of the expanded value of
    parameter, then the result of the expansion is the value of parameter
    with the shortest matching pattern (the ‘%’ case) or the longest matching
    pattern (the ‘%%’ case) deleted. If parameter is ‘@’ or ‘*’, the pattern
    removal operation is applied to each positional parameter in turn, and the
    expansion is the resultant list. If parameter is an array variable
    subscripted with ‘@’ or ‘*’, the pattern removal operation is applied to
    each member of the array in turn, and the expansion is the resultant list.
\$\endgroup\$
5
  • 2
    \$\begingroup\$ {line%% *} seems an overkill. while read -r ip rest (notice the absence of IFS) will conveniently split the line into ip and the rest. \$\endgroup\$ – vnp Sep 13 '16 at 22:03
  • \$\begingroup\$ @vpn - that's a great suggestion - and a trick I may use in the future for other things too. I have only ever used read to pull things in to a single variable before. \$\endgroup\$ – rolfl Sep 13 '16 at 22:31
  • \$\begingroup\$ echo is a bash builtin, so complaining about its use is going a bit too far. Using sort and awk within shell scripts is probably smarter than redoing those things in bash even if the OP seemed to be trying to do that. \$\endgroup\$ – chicks Sep 14 '16 at 1:06
  • \$\begingroup\$ +1 I actually thought sort, awk, grep were built in... so that's my first learning experience of the day straight away. Your code is far more readable and robust, I can follow what you've done apart from the line ip=${line%% *} can you explain that line any futher? \$\endgroup\$ – Darren H Sep 14 '16 at 4:01
  • \$\begingroup\$ @DarrenH - added references for the %% variable expander. \$\endgroup\$ – rolfl Sep 14 '16 at 10:38
4
\$\begingroup\$

First, you're making one pass to extract all of the client IP addresses. Then, for each address, you process the entire file again to obtain the hit count. For any large log file, performance is going to suck!

There are some micro-optimizations that could be done (such as removing the superfluous use of cat, and combining sort | uniq as sort -u), but that won't change the fact that this is an O(n2) solution that should be rewritten for efficiency.

My general recommendation for line-oriented processing is to use just awk, because that's what it's good at.

awk '
    /GET \/ HTTP/ { count[$1] += 1 }
    END { for (ip in counts) { print ip, "\t", counts[ip] } }
' access.log | sort -n -r -k 2

This solution uses an AWK's associative array to do all of the counting in one pass. We completely ignore all entries other than GET / HTTP requests. Each IP address and its count are printed on the same line, for easier interpretation of the results. Since, in my opinion, it doesn't do much good to sort dotted-quad IP addresses, I suggest sorting by hit count instead.

\$\endgroup\$
2
  • \$\begingroup\$ This has opened my eyes to how large awk really is, looks like I have a lot of reading to do. Can you suggest any easy-to-follow sources of information on this and similar commands? \$\endgroup\$ – Darren H Sep 14 '16 at 4:07
  • 1
    \$\begingroup\$ Is there a basic tutorial for grep, awk and sed? \$\endgroup\$ – 200_success Sep 14 '16 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.