This is a continuation from my original question. After the improvements suggested by @Reinderien and some of my own.
This approach I've taken is kinda obvious. And I'm not using any parallel processing. I think there's scope for improvement because I know of a crate, Rayon in Rust which could have run the steps I'm currently running, parallelly. I'll explain why I think this is possible below.
"""
Find the number of 'exceptions' and 'added' event's in the exception log
with respect to the device ID.
author: clmno
date: 2018-12-23
updated: 2018-12-27
"""
from time import time
import re
def timer(fn):
""" Used to time a function's execution"""
def f(*args, **kwargs):
before = time()
rv = fn(*args, **kwargs)
after = time()
print("elapsed", after - before)
return rv
return f
#compile the regex globally
re_prefix = '.*?'
re_guid='([A-Z0-9]{8}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{12})'
rg = re.compile(re_prefix+re_guid, re.IGNORECASE|re.DOTALL)
def find_sql_guid(txt):
""" From the passed in txt, find the SQL guid using re"""
m = rg.search(txt)
if m:
guid1 = m.group(1)
else:
print("ERROR: No SQL guid in line. Check the code")
exit(-1)
return guid1
@timer
def find_device_IDs(file_obj, element):
""" Find the element (type: str) within the file (file path is
provide as arg). Then find the SQL guid from the line at hand.
(Each line has a SQL guid)
Return a dict of {element: [<list of SQL guids>]}
"""
lines = set()
for line in file_obj:
if element in line:
#find the sql-guid from the line-str & append
lines.add(find_sql_guid(line))
file_obj.seek(0)
return lines
@timer
def find_num_occurences(file_obj, key, search_val, unique_values):
""" Find and append SQL guids that are in a line that contains a string
that's in search_val into 'exception' and 'added'
Return a dict of {'exception':set(<set of SQL guids>),
'added': set(<set of SQL guids>)}
"""
lines = {'exception':set(), 'added': set()}
for line in file_obj:
for value in unique_values:
if value in line:
if search_val[0] in line:
lines['exception'].add(value)
elif search_val[1] in line:
lines['added'].add(value)
file_obj.seek(0)
return lines
def print_stats(num_exceptions_dict):
for key in num_exceptions_dict.keys():
print("{} added ".format(key) +
str(len(list(num_exceptions_dict[key]["added"]))))
print("{} exceptions ".format(key) +
str(len(list(num_exceptions_dict[key]["exception"]))))
if __name__ == "__main__":
path = 'log/server.log'
search_list = ('3BAA5C42', '3BAA5B84', '3BAA5C57', '3BAA5B67')
with open(path) as file_obj:
#find every occurance of device ID and find their corresponding SQL
# guids (unique ID)
unique_ids_dict = {
element: find_device_IDs(file_obj, element)
for element in search_list
}
#Now for each unique ID find if string ["Exception occurred",
# "Packet record has been added"] is found in it's SQL guid list.
search_with_in_deviceID = ("Exception occurred",
"Packet record has been added")
#reset the file pointer
file_obj.seek(0)
num_exceptions_dict = {
elem: find_num_occurences(file_obj, elem, search_with_in_deviceID,
unique_ids_dict[elem])
for elem in search_list
}
print_stats(num_exceptions_dict)
and here's a small server log for you to experiment on
Improvements
- More pythonic with some help from Reinderien.
- Opening the file only once. Can't see a significant change in execution speed.
- Using better data structure model. Was using
dict
s everywhere,set
s made sense.
My current approach is to
- Find the device ID (eg 3BAA5C42) and their corresponding SQL GUIDs.
- For each SQL GUID find if it resulted in an
exception
oradded
event. Store them in dict. - Print the stats
Parallelize
Step one and two are just going through the file searching for a particular string and performing a set of instructions once the sting is found. And so each process (both within steps one and two, and step one and two as a whole) is independent of each other / mutually exclusive
. And so running them in parallel makes more sense.
How should I get going to improve this code?
find_device_IDs
has run for the first time, you have reached the end of the file, so all except one ID are empty sets. Afile_obj.seek(0)
at the end of the function would help for now. \$\endgroup\$ – Graipher Dec 27 '18 at 10:03def find_device_IDs(file_obj, element):
, otherwise it just accesses the global variablefile_obj
. \$\endgroup\$ – Graipher Dec 27 '18 at 10:08