3
\$\begingroup\$

I need to extract from a large XML file some information:

  • 0 or 1 attribute per tag
  • Trigger on a leaf element

So I wrote this Python code:

def ascend(element):
    parent = element.getparent()
    return [parent] + ascend(parent) if parent is not None else []

def parsexml(filename, tag, attrs):
    from lxml import etree
    for el in etree.ElementTree(file=filename).iter(tag):
        nodes = ascend(el)[::-1] + [el]
        yield [n.attrib[a] for n, a in zip(nodes,attrs) if a]

This snippet can be used as follow (with simplified data):

In [1]: xml = StringIO.StringIO("""
<data>
    <level1 a="1">
        <level2 b="1">
            <level3 c="1"/>      
            <level3 c="2"/>      
        </level2>       
        <level2 b="2">
            <level3 c="3"/>      
            <level3 c="4"/>              
        </level2>           
    </level1>
    <level1 a="2">
        <level2 b="3">
            <level3 c="5"/>      
            <level3 c="6"/>      
        </level2>       
        <level2 b="4">
            <level3 c="7"/>      
            <level3 c="8"/>              
        </level2>           
    </level1>    
</data>""")

In [2]: list(parsexml(xml, 'level3', [None, 'a', None, 'c']))
Out[2]:
[['1', '1'],
 ['1', '2'],
 ['1', '3'],
 ['1', '4'],
 ['2', '5'],
 ['2', '6'],
 ['2', '7'],
 ['2', '8']]

My goal is to import the extracted data in Pandas:

ds = pd.DataFrame(data, columns=['level1', 'level3'])

Is my implementation correct or can I do better? I think the code is short and elegant, but I feel it spends most of its time looking for parents which can be ineffective (I didn't measured it and for my application it is still fast enough).

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.