I'm aware I am probably reinventing the wheel somewhat here, but I am trying to teach myself simple bash coding by completing simple tasks such as parsing files.

To that end I am looking to learn what elephants in the room I could be missing or if there are better ways to use the core functionality of bash without installing any additional tools.

This simple code returns a list of unique IP addresses that have hit the index of my site along with a count of the hits.

a="access.log"; for b in $(cat $a | awk '{print $1}' | sort | uniq);do echo $b;grep $a -e "GET / HTTP" | grep -c $b;done;

Assumptions: access.log is in the current directory and is in the regular format

Any advice or suggestions for improvement are greatly appreciated


2 Answers 2


Well, your code is hardly a bash solution, is it? You use sort, awk, grep, and echo....

Additionally, your code is dumped on a single line, and it makes it hard to read. Why not put it in a script, and have separate commands on separate lines.... like:


for b in $(cat $a | awk '{print $1}' | sort | uniq); do
  echo $b;
  grep $a -e "GET / HTTP" | grep -c $b;

Those variable names.... ouch. a and b make it hard to separate from the -c and -e too.... and they mean nothing otherwise. Why not use meaningful names like ip and log?

Then, when I ran the code, I got a lot of funny results.... like:

Why are there 0 counts.... oh, that's because those are IP's that are not accessing the home page, but are accessing other pages... they appear as $b but don't actually "GET" /.

I would consider making it more a study of bash and use the native bash structures to get things right.... no grep, awk, etc.


# use first commandline argument if supplied
if [ $1 ] ; then

# set a variable to match in a regular expression
match="GET / HTTP"

# create a named array.
declare -A counts

# read the file line-by-line
while IFS='' read -r line || [[ -n "$line" ]]; do

  # find lines that access GET / HTTP
  if [[ $line =~ $match ]] ; then

    # get just the IP of the client
    ip=${line%% *}
    # get the previous count, default to 0
    # increment the count for this IP
    counts[$ip]=$(($prev + 1))
done < "$log"

for ip in "${!counts[@]}" ; do
    echo "IP $ip visited ${counts[$ip]} times"

EDIT: About the ${line%% *} variable substitution. The possibilities when doing variables in bash are remarkably powerful. I recommend looking at the document Parameter Substitution for details, and the man page for is good as well (but does not have the examples). The %% token indicates that there should be a pattern search backward from the end of $line for a space followed by any characters (the * - this is a "glob" expression, not a regex). This pattern essentially looks for the first space, and removes it and any charaters after it. The man page document says:


    The word is expanded to produce a pattern just as in filename expansion.
    If the pattern matches a trailing portion of the expanded value of
    parameter, then the result of the expansion is the value of parameter
    with the shortest matching pattern (the ‘%’ case) or the longest matching
    pattern (the ‘%%’ case) deleted. If parameter is ‘@’ or ‘*’, the pattern
    removal operation is applied to each positional parameter in turn, and the
    expansion is the resultant list. If parameter is an array variable
    subscripted with ‘@’ or ‘*’, the pattern removal operation is applied to
    each member of the array in turn, and the expansion is the resultant list.
  • 2
    \$\begingroup\$ {line%% *} seems an overkill. while read -r ip rest (notice the absence of IFS) will conveniently split the line into ip and the rest. \$\endgroup\$
    – vnp
    Sep 13, 2016 at 22:03
  • \$\begingroup\$ @vpn - that's a great suggestion - and a trick I may use in the future for other things too. I have only ever used read to pull things in to a single variable before. \$\endgroup\$
    – rolfl
    Sep 13, 2016 at 22:31
  • \$\begingroup\$ echo is a bash builtin, so complaining about its use is going a bit too far. Using sort and awk within shell scripts is probably smarter than redoing those things in bash even if the OP seemed to be trying to do that. \$\endgroup\$
    – chicks
    Sep 14, 2016 at 1:06
  • \$\begingroup\$ +1 I actually thought sort, awk, grep were built in... so that's my first learning experience of the day straight away. Your code is far more readable and robust, I can follow what you've done apart from the line ip=${line%% *} can you explain that line any futher? \$\endgroup\$
    – Darren H
    Sep 14, 2016 at 4:01
  • \$\begingroup\$ @DarrenH - added references for the %% variable expander. \$\endgroup\$
    – rolfl
    Sep 14, 2016 at 10:38

First, you're making one pass to extract all of the client IP addresses. Then, for each address, you process the entire file again to obtain the hit count. For any large log file, performance is going to suck!

There are some micro-optimizations that could be done (such as removing the superfluous use of cat, and combining sort | uniq as sort -u), but that won't change the fact that this is an O(n2) solution that should be rewritten for efficiency.

My general recommendation for line-oriented processing is to use just awk, because that's what it's good at.

awk '
    /GET \/ HTTP/ { count[$1] += 1 }
    END { for (ip in counts) { print ip, "\t", counts[ip] } }
' access.log | sort -n -r -k 2

This solution uses an AWK's associative array to do all of the counting in one pass. We completely ignore all entries other than GET / HTTP requests. Each IP address and its count are printed on the same line, for easier interpretation of the results. Since, in my opinion, it doesn't do much good to sort dotted-quad IP addresses, I suggest sorting by hit count instead.

  • \$\begingroup\$ This has opened my eyes to how large awk really is, looks like I have a lot of reading to do. Can you suggest any easy-to-follow sources of information on this and similar commands? \$\endgroup\$
    – Darren H
    Sep 14, 2016 at 4:07
  • 1
    \$\begingroup\$ Is there a basic tutorial for grep, awk and sed? \$\endgroup\$ Sep 14, 2016 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.