What does it do ?
Looks in big XML files (1-3 GB) for given parameters that I need in my project, appends them to lists and finally exports both of them to the CSV files.
My XML scheme
I need to get child named 'v', which is value of specific child 'k' nested in tag named 'tag' which is nested within parent 'way'.
What I tried
Plus various scripts from StackExchange.
Performance
My function needs 50% less time to parse XML than above method, any other I tried or I was able to try, because with some I was unable to figure things out and given up on them.
My goal
To get some tips on my code, hopefully find way to speed it up. I need to parse about 20 - 30 GB of XML files (as mentioned before single file is 1-3 GB), whole procedure is really time consuming - It takes about 12 hours up to 2 days of non stop parsing.
Variables description:
- xml - path to xml file
- list_agis - list of IDs, list which length is number between 5k - 1 kk
- parent - way
- child - tag
- child_atribitute - k
- child_value_1 - highway
- child_value_2 - track
- name_id, name_atribute, name_file, path_csv, part_name - variables needed to create csv file name
My code
def save_to_csv(
list_1, list_2, name_1,
name_2, csv_name, catalogue,
part_name):
"""
Saves to CSV, based on 2 lists.
"""
raw_data = {name_1: list_1,
name_2: list_2}
df = pd.DataFrame(raw_data, columns=[name_1, name_2])
df.to_csv(
'{0}\{1}_{2}.csv'.format(catalogue, part_name, csv_name),
index=False, header=True, encoding = 'CP1250')
def xml_parser(
xml, lista_agis, parent,
atribiute_parent, child, child_atribiute,
child_value_1, child_value_2, name_file,
sciezka_csv, name_id, name_atribiute,
part_name):
"""
Function to pick from xml files tag values.
Firstly it creates tree of xml file and then
goes each level town and when final condtion is fullfiled
id and value from xml file is appended to list in the end of
xml file list is saved to CSV.
"""
rootElement = ET.parse(xml).getroot()
list_id = []
list_value = []
for subelement in rootElement:
if subelement.tag == parent:
if subelement.get(atribiute_parent) in lista_agis:
for sselement in subelement:
if sselement.tag == child:
if sselement.attrib[child_atribiute] == child_value_1:
list_id.append(
subelement.get(atribiute_parent))
list_value.append(
sselement.get(child_value_2))
save_to_csv(
list_id, list_value, name_id,
name_atribiute, name_file,
sciezka_csv, part_name)
k
andv
are "attributes", not "children". \$\endgroup\$ – Reinderien Nov 21 '18 at 16:55