Search for ways in OpenStreetMap extract

Question

What does it do ?

Looks in big XML files (1-3 GB) for given parameters that I need in my project, appends them to lists and finally exports both of them to the CSV files.

My XML scheme

I need to get child named 'v', which is value of specific child 'k' nested in tag named 'tag' which is nested within parent 'way'.

What I tried

IBM's take on big xml files

Plus various scripts from StackExchange.

Performance

My function needs 50% less time to parse XML than above method, any other I tried or I was able to try, because with some I was unable to figure things out and given up on them.

My goal

To get some tips on my code, hopefully find way to speed it up. I need to parse about 20 - 30 GB of XML files (as mentioned before single file is 1-3 GB), whole procedure is really time consuming - It takes about 12 hours up to 2 days of non stop parsing.

Variables description:

• xml - path to xml file
• list_agis - list of IDs, list which length is number between 5k - 1 kk
• parent - way
• child - tag
• child_atribitute - k
• child_value_1 - highway
• child_value_2 - track
• name_id, name_atribute, name_file, path_csv, part_name - variables needed to create csv file name

My code

def save_to_csv(
        list_1, list_2, name_1, 
        name_2, csv_name, catalogue, 
        part_name):
    """
    Saves to CSV, based on 2 lists.
    """
    raw_data = {name_1: list_1,
                name_2: list_2}
    df = pd.DataFrame(raw_data, columns=[name_1, name_2])
    df.to_csv(
        '{0}\{1}_{2}.csv'.format(catalogue, part_name, csv_name), 
        index=False, header=True, encoding = 'CP1250')


def xml_parser(
        xml, lista_agis, parent, 
        atribiute_parent, child, child_atribiute,
        child_value_1, child_value_2, name_file,
        sciezka_csv, name_id, name_atribiute, 
        part_name):
    """
    Function to pick from xml files tag values.
    Firstly it creates tree of xml file and then
    goes each level town and when final condtion is fullfiled
    id and value from xml file is appended to list in the end of
    xml file list is saved to CSV.

    """
    rootElement = ET.parse(xml).getroot()
    list_id = []
    list_value = []
    for subelement in rootElement:
        if subelement.tag == parent:
            if subelement.get(atribiute_parent) in lista_agis:
                for sselement in subelement:
                    if sselement.tag == child:
                        if sselement.attrib[child_atribiute] == child_value_1:
                            list_id.append(
                                subelement.get(atribiute_parent))
                            list_value.append(
                                sselement.get(child_value_2))

    save_to_csv(
        list_id, list_value, name_id, 
        name_atribiute, name_file, 
        sciezka_csv, part_name)

Answer 1

Introduction

Unfortunately imposm is just for Python 2, my project is in Python 3. I think lxml library looks promising. I wrote simple code to test it, right now it is based on just 2 nodes.

Picture of nodes

I attach picture from XML file so one can see what I am dealing with.

How it works

It is iterating over l_id (which is list of ids) using etree.parse and findall.

First inner loop gathers dictionary where given id is.

Second inner loop gathers dictionary where chosen value is.

Loop for dict_ids_all appends to new list only ids from dictionary.

Loop for dict_ids_all appends to new list only value from dictionary.

My code

tree = lxml.etree.parse(r'path to xml')
l_dict_ids_all = []
l_dict_values_all= []
l_only_id =[]
l_only_values = []
l_id = ['"35121262"', '"35121263"']
name = '"name"'

for id in l_id:
    for tag in tree.findall('//node[@id={0}]'.format(id)):
        l_dict_ids_all.append(tag.attrib)

    for tag in tree.findall('//node[@id={0}]//tag[@k={1}]'.format(id,name)):
        l_dict_values_all.append(tag.attrib)


#printing is only for review purpose

print('Full id dict')        
print(l_dict_ids_all)

print('Full Value dict')
print(l_dict_values_all)

print('Only ID list')
for element in l_dict_ids_all:
    l_only_id.append(element['id'])
print(l_only_id)

print('Only Value list')
for element in l_dict_values_all:
    l_only_values.append(element['k'])
print(l_only_values)

Output

Full id dict

[{'id': '35121262', 'visible': 'true', 'version': '17', 'changeset': '41419943', 'timestamp': '2016-08-12T22:24:23Z', 'user': 'kocio', 'uid': '52087', 'lat': '52.1560439', 'lon': '21.0346808'}, {'id': '35121263', 'visible': 'true', 'version': '16', 'changeset': '41419943', 'timestamp': '2016-08-12T22:24:20Z', 'user': 'kocio', 'uid': '52087', 'lat': '52.1492285', 'lon': '21.0461042'}]

Full Value dict [{'k': 'name', 'v': 'Stokłosy'}, {'k': 'name', 'v': 'Imielin'}]

Only ID list ['35121262', '35121263']

Only Value list ['name', 'name']

What I tried

I am aware that creating list and using it to append items to new list is wrong, but whenever I tried something like this:

l_dict_ids_all.append(tag.attrib[0]['id'])

Received an error :

TypeError Traceback (most recent call)

ipython-input-91-8b0a49bc5f35 in () 7 for id in l_id: 8 for tag in tree.findall('//node[@id={0}]'.format(id)): ----> 9 l_dict_ids_all.append(tag.attrib[0]['id']) 10 src/lxml/etree.pyx in lxml.etree._Attrib.getitem()

src/lxml/apihelpers.pxi in lxml.etree._getAttributeValue()

src/lxml/apihelpers.pxi in lxml.etree._getNodeAttributeValue()

src/lxml/apihelpers.pxi in lxml.etree._getNsTag()

src/lxml/apihelpers.pxi in lxml.etree.__getNsTag()

src/lxml/apihelpers.pxi in lxml.etree._utf8()

TypeError: Argument must be bytes or unicode, got 'int'

My goal/problem

Code is working, but I want to make it better. I need to get rid of 2 out 4 lists which I create at the begging.