Description
Simply take a JSON file as input and convert the data in it into a CSV file. I won't describe the functionality in too much detail since I have reasonable docstrings for that. As you can see, my solution is not memory efficient since I'm reading all the file into memory.
I'd like to improve the performance of my solution as much as possible. (perhaps not load everything at once into memory -- even if it's gonna be slower).
The JSON file that I'm trying to convert is 60 GB and I have 64GB of RAM.
Code
import csv
import json
CSV_PATH = 'file.csv'
JSON_PATH = 'file.json'
def flattenjson(json_data, delim):
"""
Flatten a simple JSON by prepending a delimiter to nested children.
Arguments:
json_data (dict): JSON object
e.g: {
"key1": "n1_value1",
"key2": "n1_value2",
"parent1": {
"child_key1": "n1_child_value1",
"child_key2": "n1_child_value2"
}
}
delim (str): Delimiter for nested children (e.g: '.')
Returns:
Flattened JSON object.
e.g: {
'key1': 'n1_value1',
'key2': 'n1_value2',
'parent1.child_key1': 'n1_child_value1',
'parent1.child_key2': 'n1_child_value2'
}
"""
flattened_json = {}
for i in json_data.keys():
if isinstance(json_data[i], dict):
get = flattenjson(json_data[i], delim)
for j in get.keys():
flattened_json[i + delim + j] = get[j]
else:
flattened_json[i] = json_data[i]
return flattened_json
def write_json_to_csv(flattened_json, csv_path):
"""
Write flattened json to a csv file. The keys of the json will be the header
of the csv and the values..well, the values ^_^.
Arguments:
flattened_json (dict): Flattened JSON object.
e.g: {
'key1': 'n1_value1',
'key2': 'n1_value2',
'parent1.child_key1': 'n1_child_value1',
'parent1.child_key2': 'n1_child_value2'
}
csv_path (str): path of the CSV file
Returns:
None
"""
with open(csv_path, 'w') as out_file:
w = csv.DictWriter(out_file, flattened_json.keys())
w.writeheader()
w.writerow(flattened_json)
def main():
"""
Main entry to our program.
"""
with open(JSON_PATH) as json_file:
json_data = json.load(json_file)
flattened_json = flattenjson(json_data, '.')
write_json_to_csv(flattened_json, CSV_PATH)
if __name__ == '__main__':
main()
More about input / output
- I don't know where the JSON file comes from so I have to stick with it and process it as is.
- I can't change the structure of the JSON file
- As far I saw, the JSON data will be at most 7 level-nested so we could have something like:
{
"a": "1",
"b": "2",
"c": {
"c_1": "3",
"c_2": "4"
},
"d": {
"d_1": {
"d_1_1": "5",
"d_1_2": "6"
},
"d_2": {
"d_2_1": "5",
"d_2_2": "6"
}
... and so on
}
}
- I have to write the data to the CSV file as described above.
- The CSV for the above JSON will look like this:
I'm specifically looking for a review orientated on memory optimizations which probably comes with a cost of a slower running time (that's fine) but any other overall improvements are welcome!
PS: I've done the above in Python 3.8.2 so I'd like you to focus on a version of Python >= 3.6