6
\$\begingroup\$

I use the following command to get specific entries between certain dates from Apache log files. Then I filter out my own address and some bots, then I print out the IP, sort it, get the uniques, output the ip with count, and for summary I count lines to get a total.

Is there a way to improve this, reducing the amount of awk commands? Or to get this process down more efficient? I'm not that proficient with awk, but I have the feeling this should be accomplish able in maybe one awk instruction.

zgrep '"GET /my/path/to/page.html' other_vhosts_access* 
| awk -F'[][]' 
    -v dstart=`date -d"2019-12-09" +%Y%m%dT%0H:%0M:%0S` 
    -v dend=`date -d"2020-01-09" +%Y%m%dT%0H:%0M:%0S` 
    '{ $2 = substr($2,8,4)sprintf("%02d",(match("JanFebMarAprMayJunJulAugSepOctNovDec",substr($2,4,3))+2)/3)substr($2,1,2)"T"substr($2,13,8); 
    if ($2 >= dstart && $2 < dend) print }' 
| awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/' 
| awk '{print $2}' 
| sort 
| uniq -c 
| sort -n 
| wc -l

the one liner:

zgrep '"GET /my/path/to/page.html' other_vhosts_access* | awk -F'[][]' -v dstart=`date -d"2019-12-09" +%Y%m%dT%0H:%0M:%0S` -v dend=`date -d"2020-01-09" +%Y%m%dT%0H:%0M:%0S` '{ $2 = substr($2,8,4)sprintf("%02d",(match("JanFebMarAprMayJunJulAugSepOctNovDec",substr($2,4,3))+2)/3)substr($2,1,2)"T"substr($2,13,8); if ($2 >= dstart && $2 < dend) print }' | awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/' | awk '{print $2}' | sort | uniq -c | sort -n | wc -l
\$\endgroup\$
1
  • 2
    \$\begingroup\$ A small sample of the log file would be useful for testing and also validating your logic. \$\endgroup\$
    – tripleee
    Commented Jan 10, 2020 at 18:37

3 Answers 3

5
\$\begingroup\$

Here's a refactoring which condenses much of the logic after the zgrep into a single Awk script.

  • Prefer modern $(command substitution) over obsolescent `command substitution`syntax. But the date command substitutions don't really make sense here. Just pass in the dates as literal strings, like dstart="20190109T00:00:00".
  • Two Awk processes can often be merged. If you have awk -F : '{print $2}' | awk -F = '{print $1}' you can simply use awk -F : '{ split($2, x, /=/); print x[1] }'
  • I refactored the bot\.|bot\/ fragment in the regex to bot[.\/].
  • uniq -c can be replaced with a simple Awk associative array of counts. This does away with the first sort.
  • If you only care about the number of unique IP addresses, there is no need for sort -n, and vice versa. I'm guessing you want either, so have not attempted to replace those parts.
  • And of course, don't introduce syntax errors when splitting the script over multiple lines. A pipe | at end of line naturally splits the script, while a newline followed by a pipe is an error.
zgrep '"GET /my/path/to/page.html' other_vhosts_access* |
awk -F'[][]' \
    -v dstart="20191209T00:00:00" -v dend="20200109T00:00:00" \
    '{ $2 = substr($2,8,4) sprintf("%02d",(\
            match("JanFebMarAprMayJunJulAugSepOctNovDec",
            substr($2,4,3))+2)/3) substr($2,1,2) "T" substr($2,13,8); 
       if ($2 >= dstart && $2 < dend && \
           $0 !~ /127\.0\.0\.1|bot[.\]/|dotbot|crawler/) {
        split($0, x, /[ \t]+/)
        ip=x[2]
        ++p[ip] }
  }
  END { for (ip in p) printf "%7i %s\n", p[ip], ip }' |
# sort -n |
wc -l

If you always want only the total, the end of the script can be replaced with

        if (!p[ip]++) total++ }
    }
    END { print total }'

but if you sometimes want to see individual IP addresses, I would just keep the option to pipe to either sort -n or wc -l.

Kudos for the rather compact date extraction logic. Of course, if Apache didn't default to a horrible "human readable" date format, this would not be necessary; but I guess we are stuck with it.

\$\endgroup\$
0
7
\$\begingroup\$

I assume there's a simply copy-paste error where you have | at the start of a line, instead of at the end of the preceding line?

If we're just going to count lines, there's no need for sort | uniq -c | sort -n - we can replace all that with sort -u.

These two Awk programs can be trivially combined:

awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/' | awk '{print $2}'

That would become

awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/ {print $2}'
\$\endgroup\$
6
  • \$\begingroup\$ I guess the remaining two Awk scripts can also be combined; just split() the value of ` $2` and print the second element from the resulting array if $2 matches the regex. \$\endgroup\$
    – tripleee
    Commented Jan 10, 2020 at 15:21
  • \$\begingroup\$ @tripleee, I'm only an occasional Awk user; I recommend that you make that your answer. If you really don't want to write your own answer, then edit mine to add your recommendation (but I'd prefer you to write your own!) \$\endgroup\$ Commented Jan 10, 2020 at 15:25
  • \$\begingroup\$ There is no copy paste error. I basically used the pipes as a newline to make the scrip easier to read. All the pipes are also in the oneliner. The newlines are just there for the ease of reading. \$\endgroup\$ Commented Jan 10, 2020 at 16:24
  • \$\begingroup\$ zgrep '"GET /path/to/page HTTP' other_vhosts_access* | awk -F'[][]' -v dstart=``date -d"2019-12-09" +%Y%m%dT%0H:%0M:%0S`` -v dend=``date -d"2020-01-09" +%Y%m%dT%0H:%0M:%0S`` '{ $2 = substr($2,8,4)sprintf("%02d",(match("JanFebMarAprMayJunJulAugSepOctNovDec",substr($2,4,3))+2)/3)substr($2,1,2)"T"substr($2,13,8); if ($2 >= dstart && $2 < dend) print }' | awk '$0 !~ /127\.0\.0\.1|bot\.|bot\/|dotbot|crawler/ {print $2}' | sort -u | wc -l works great for counting lines. But if I wish to remove the wc -l part to get an overview of how much the same ip requested I still need to use the extras \$\endgroup\$ Commented Jan 10, 2020 at 16:32
  • 1
    \$\begingroup\$ Thanks for prodding me! Posted a separate answer now. \$\endgroup\$
    – tripleee
    Commented Jan 10, 2020 at 18:38
3
\$\begingroup\$

Drop the unnecessary time part

The time part of the date doesn't play a role in the filtering. You could drop it and the script will be simpler.

Use more variables

I find this line a bit difficult to read for multiple reasons:

'{ $2 = substr($2,8,4)sprintf("%02d",(match("JanFebMarAprMayJunJulAugSepOctNovDec",substr($2,4,3))+2)/3)substr($2,1,2)"T"substr($2,13,8); 

First of all, I don't see a good reason to overwrite the original value of the field variable $2. It would be natural to store the computed date string in a variable called date.

Secondly, the elements of the formatting logic are hard to read, and it would be easy to fix by assigning them to variables like year, month, day.

Finally, a space between the concatenated elements such as substr(...)sprintf(...) would be really welcome by human readers.

Use $(...) instead of `...`

There's really no reason to use the archaic and potentially troublesome `...` syntax.

Order terms in a condition by value

Instead of this:

if ($2 >= dstart && $2 < dend) ...

Consider this:

if (dstart <= $2 && $2 < dend) ...

When the values increase from left to right, the meaning becomes natural, intuitive.

Don't repeat yourself

Instead of typing +%Y%m%dT%0H:%0M:%0S twice, it would be better to write it once and store it in a variable.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.