# Filtering file for command line tool

I've written a little command-line utility for filtering log files. It's like grep, except instead of operating on lines it operates on log4j-style messages, which may span multiple lines, with the first line always including the logging level (TRACE, DEBUG etc.).

Example usage on a file short.log with contents like this:

16:16:12 DEBUG - Something happened, here's a couple lines of info:
debug line
another debug line
16:16:14 - I'm being very verbose 'cause you've put me on TRACE
trace info
16:16:15 TRACE - single line trace
16:16:16 DEBUG - single line debug


logrep -f short.log DEBUG produces:

16:16:12 DEBUG - Something happened, here's a couple lines of info:
debug line
another debug line
16:16:16 DEBUG - single line debug


I think the main loop of the program could probably be simplified with some sort of parse and filter.

  file = fileinput.input(options.file)
try:
line = file.next()
while True:
if any(s in line for s in loglevels):
if filter in line:
sys.stdout.write(line)
line = file.next()
while not any(s in line for s in loglevels):
sys.stdout.write(line)
line = file.next()
continue
line = file.next()
except StopIteration:
return

• Shouldn't you consider the log level only if it is at the beginning of the line? A line could contain info without being the first line of an info log. (Also you should log on a single line and use other separators if its spans on multiple "lines" but that is just my point of view because I handle huge quantities of logs) Sep 24, 2013 at 22:45
• The level isn't at the beginning of the line, it's after a timestamp. I could parse it to ensure the level is where I expect it to be, but I'm trying to be robust to minor changes in logging format. Sep 24, 2013 at 23:17

I see two problems with your loops.

In terms of style, calling file.next() and catching StopIteration is highly unconventional. The normal way to iterate is:

for line in fileinput.input(options.file):
…


In terms of functionality, I would personally consider the grepper to be buggy because it will fail to find a log message where the filter keyword that you are seeking appears on a continuation line.

To solve both problems, I would decompose the problem into two parts: reconstructing the logical messages (somewhat ugly) and searching (relatively straightforward).

import fileinput
import re

def log_messages(lines):
"""
Given an iterator of log lines, generate pairs of
(level, message), where message is a logical log message.
possibly multi-line.
"""
log_level_re = re.compile(r'\b(TRACE|DEBUG|WARN|ERROR|CRITICAL)\b')
message = None
for line in lines:
match = log_level_re.search(line)
if match:                               # First line
if message is not None:
yield level, message
level, message = match.group(), line
elif message is not None:               # Continuation line
message += line
if message is not None:                     # End of file
yield level, message

for level, message in log_messages(fileinput.input(options.file)):
if filter in message:
sys.stdout.write(message)


Note that I've used a regular expression to look for TRACE, DEBUG, etc. The \b anchors ensure that we don't mistake words like "INTRACELLULAR" for a TRACE message.