The code below takes a directory of XML files and parses them into a CSV file. Right now parsing around 60 XML files is fast and the output is a CSV file that is around 250MB.
That is a really big file and the reason is because columns are being repeated. I'm repeating the columns for the reason that every element should have all the information. In red is one of the cases where the ID Z048 had multiple lines of
setdata so that is why the other columns in red had to be repeated.
I'm planning to increase the number of XML files to 5k, meaning that the CSV file will be relatively large.
Asking this question to maybe get any answer if the size of my CSV file can be lowered somehow. Even though I tried to code with the mindset that I want my code to be fast and produce not too big CSV files.
from xml.etree import ElementTree as ET from collections import defaultdict import csv from pathlib import Path directory = 'path to a folder with xml files' with open('output.csv', 'w', newline='') as f: writer = csv.writer(f) headers = ['id', 'service_code', 'rational', 'qualify', 'description_num', 'description_txt', 'set_data_xin', 'set_data_xax', 'set_data_value', 'set_data_x'] writer.writerow(headers) xml_files_list = list(map(str, Path(directory).glob('**/*.xml'))) print(xml_files_list) for xml_file in xml_files_list: tree = ET.parse(xml_file) root = tree.getroot() start_nodes = root.findall('.//START') for sn in start_nodes: row = defaultdict(str) repeated_values = dict() for k,v in sn.attrib.items(): repeated_values[k] = v for rn in sn.findall('.//Rational'): repeated_values['rational'] = rn.text for qu in sn.findall('.//Qualify'): repeated_values['qualify'] = qu.text for ds in sn.findall('.//Description'): repeated_values['description_txt'] = ds.text repeated_values['description_num'] = ds.attrib['num'] for st in sn.findall('.//SetData'): for k,v in st.attrib.items(): row['set_data_'+ str(k)] = v for key in repeated_values.keys(): row[key] = repeated_values[key] row_data = [row[i] for i in headers] writer.writerow(row_data) row = defaultdict(str)