3
\$\begingroup\$

I made the following regular expression for parsing ngnix log

log_1 = "1.169.137.128 -  - [29/jun/2017:07:10:50 +0300] "GET /api/v2/banner/1717161 http/1.1" 200 2116 "-" "Slotovod" "-" "1498709450-2118016444-4709-10027411" "712e90144abee9" 0.199"

My test cases (https://regex101.com/r/Eyhxod/1)

lineformat = re.compile(r"""(?P<ipaddress>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) -  - \[(?P<dateandtime>\d{2}\/[a-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2} (\+|\-)\d{4})\] \"GET (?P<url>.+?(?=\ http\/1.1")) http\/1.1" \d{3} \d+ "-" (?P<http_user_agent>.+?(?=\ )) "-" "(?P<x_forwaded_for>(.+?))" "(?P<http_xb_user>(.+?))" (?P<request_time>[+-]?([0-9]*[.])?[0-9]+)""",re.IGNORECASE)

Output:

data = re.search(lineformat, log_1)
data.groupdict()

{'ipaddress': '1.169.137.128',
 'dateandtime': '29/jun/2017:07:10:50 +0300',
 'url': '/api/v2/banner/1717161',
 'http_user_agent': '"Slotovod"',
 'x_forwaded_for': '1498709450-2118016444-4709-10027411',
 'http_xb_user': '712e90144abee9',
 'request_time': '0.199'}

I believe I should make it more robust towards edge cases and broken logs. Also I consider splitting my long expression into a smaller one. Any advices towards the best-practices are appreciated.

\$\endgroup\$
  • 1
    \$\begingroup\$ Isn't there a commonly known Python module for parsing these log files? You are for sure not the first person in the world to try this. \$\endgroup\$ – Roland Illig Mar 5 at 6:29
1
\$\begingroup\$

At the very least, use verbose mode so you can see the whole thing at once. Remember to explicitly include whitespace.

lineformat = re.compile(r"""
   (?P<ipaddress>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+
   -\s+
   -\s+
   \[(?P<dateandtime>\d{2}\/[a-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2} (\+|\-)\d{4})\]\s+
   \"GET (?P<url>.+?(?=\ http\/1.1")) http\/1.1"\s+
   \d{3}\s+
   \d+\s+
   "-"\s+
   (?P<http_user_agent>.+?(?=\ ))\s+
   "-"\s+
   "(?P<x_forwaded_for>(.+?))"\s+
   "(?P<http_xb_user>(.+?))"\s+
   (?P<request_time>[+-]?([0-9]*[.])?[0-9]+)
   """,
   re.IGNORECASE | re.VERBOSE)
| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.