This is a follow-up to https://stackoverflow.com/questions/71194832/converting-json-based-log-into-column-format-i-e-one-file-per-column .
My task is to optimize this code whose function is to convert the content of this JSON-formatted log file to columnar-formatted .txt
file for each column. An example for a log file is:
{"timestamp": "2022-01-14T00:12:21.000", "Field1": 10, "Field_Doc": {"f1": 0}}
{"timestamp": "2022-01-18T00:15:51.000", "Field_Doc": {"f1": 0, "f2": 1.7, "f3": 2}}
It will generate 5 files:
timestamp.column
Field1.column
Field_Doc.f1.column
Field_Doc.f2.column
Field_Doc.f3.column
The column file format is as follows:
- string fields are separated by a new line
'\n'
character. Assume that no string value contains new line characters, so no need to worry about escaping them - double, integer & boolean fields are represented as a single value per line
- null, undefined & empty strings are represented as an empty line
Example content of timestamp.column:
2022-01-14T00:12:21.000
2022-01-18T00:15:51.000
Note: The fields in the log will be dynamic, do not assume that these are the expected properties.
My current code for this is:
import json
import os
def flatten_dict(data, prefix=""):
result = {}
for key, value in data.items():
if prefix:
key = prefix + "." + key
if isinstance(value, dict):
result.update( flatten_dict(value, key) )
else:
if value is None:
result[key] = "\n"
elif value is "":
result[key] = "\n"
else:
result[key] = value
return result
path = input("Enter the path for the log file: ")
# Checking if path exists to the selected log
assert os.path.exists(path), "I did not find the file at, "+str(path)
file_obj = open(path) # emulate file in memory
for line in file_obj:
data = json.loads(line)
data = flatten_dict(data)
for key, value in data.items():
with open(key + '.column', "a") as f:
f.write(str(value) + "\n")
This code is currently very slow. It takes about 45 min just to parse 256mb of data which should have taken about 30sec or max a minute.
Can anyone guide me how can I optimize the efficiency? Also how can I print the usage of CPU and RAM for the same? File size to parse may vary up to 2 GB.
Any help would be appreciated.