Parsing XML to get all elem.tag: elem.text pairs

Question

I am parsing ICD-10 codes and have achieved a result that satisfies my base case, but I'm worried about how fragile my method is:

import xml.etree.ElementTree as ET

# A sample of the larger XML file I'm parsing
data = '''<diag>
<name>A00</name>
<desc>Cholera</desc>
<diag>
  <name>A00.0</name>
  <desc>Cholera due to Vibrio cholerae 01, biovar cholerae</desc>
  <inclusionTerm>
    <note>Classical cholera</note>
  </inclusionTerm>
</diag>
<diag>
  <name>A00.1</name>
  <desc>Cholera due to Vibrio cholerae 01, biovar eltor</desc>
  <inclusionTerm>
    <note>Cholera eltor</note>
  </inclusionTerm>
</diag>
<diag>
  <name>A00.9</name>
  <desc>Cholera, unspecified</desc>
</diag>
</diag>'''

# Create the tree
tree = ET.ElementTree(ET.fromstring(data))

# the `iter` method returns all tags in a given tree
# I'm just grouping the tags and texts here
def get_all_elements(tree):
    return [(elem.tag, elem.text) for elem in tree.iter()]

# This will return the desired elements from my tree
def parse_elements(tree):

    # First, get all of the elements in the tree
    elements = get_all_elements(tree)

    to_return = {}

    # This is what I think is too fragile. I'm basically looking
    # ahead for each element to see whether the next two elements
    # match the diag -> name -> desc sequence. But this indexing seems
    # to be too fragile at scale.
    for idx, elem in enumerate(elements):
        if 'diag' in elem and 'name' in elements[idx+1] and 'desc' in elements[idx+2]:
            name = elements[idx+1]
            desc = elements[idx+2]
            to_return[name[1]] = desc[1]
    return to_return

res = parse_elements(tree)

Let's see what's in res:

for k,v in res.items():
    print("name (code): ", k, "\n  desc: ", v)

name (code):  A00.9 
  desc:  Cholera, unspecified
name (code):  A00.0 
  desc:  Cholera due to Vibrio cholerae 01, biovar cholerae
name (code):  A00 
  desc:  Cholera
name (code):  A00.1 
  desc:  Cholera due to Vibrio cholerae 01, biovar eltor

So, I achieve my desired output, but I keep thinking there much be a better way to parse this XML. Unfortunately there's no <diag><subdiag></subdiag></diag>-type of hierarchy. Even subdiagnoses are labeled with the <diag> tag. I suppose the above is my attempt at recursion -- although I have found true recursion tough with the current tag-naming. At the end of the day, I just need the name: desc pairs.

Edit:

There's also this approach:

names = [x.text for x in ET.fromstring(data).findall('.//name')]
descs = [x.text for x in ET.fromstring(data).findall('.//desc')]

res = zip(names, descs)

But I don't think this approach scales well, as the number of elements in names and descs differed by about 1000 when I tested this on the larger XML file. There were mismatches between codes and descriptions when I validated on actual codes.

Answer 1

Since the latter approach resulted into broken mismatched pairs, looks like there are codes with no descriptions.

If you want to capture only the diagnosis codes that have descriptions (existing desc nodes), you can enforce this rule with the following XPath expression:

.//diag[name and desc]

The problem though is that xml.etree.ElementTree supports a limited set of XPath features and for this particular expression to work, you need to switch to lxml.etree. But, it will come with a performance boost, better memory usage and a richer functionality. It's worth it.

You can also simplify the way you extract codes by using findtext() and a dictionary comprehension:

from pprint import pprint

import lxml.etree as ET


data = """your XML here"""
root = ET.fromstring(data)

result = {diag.findtext("name"): diag.findtext("desc")
          for diag in root.xpath(".//diag[name and desc]")}

pprint(result)

Answer 2

XPath support fetching parent element https://docs.python.org/2/library/xml.etree.elementtree.html#supported-xpath-syntax. Let's try to find name element and get parent of this element:

>>> [x for x in tree.findall('.//name/..')]
[<Element 'diag' at 0x7f9c8cece278>,
 <Element 'diag' at 0x7f9c85a84908>,
 <Element 'diag' at 0x7f9c85a93f98>,
 <Element 'diag' at 0x7f9c85a8f188>]

When we have parent element, we can get name and desc elements:

>>> [(x.find('name'), x.find('desc')) for x in tree.findall('.//name/..')]
[(<Element 'name' at 0x7f9c875ba7c8>, <Element 'desc' at 0x7f9c8cedfe58>),
 (<Element 'name' at 0x7f9c85a84958>, <Element 'desc' at 0x7f9c85a84f98>),
 (<Element 'name' at 0x7f9c85a8f048>, <Element 'desc' at 0x7f9c85a8f098>),
 (<Element 'name' at 0x7f9c85a8f1d8>, <Element 'desc' at 0x7f9c85a8f228>)]

And finally:

>>> [(x.find('name').text, x.find('desc').text) for x in tree.findall('.//name/..')]
[('A00', 'Cholera'),
 ('A00.0', 'Cholera due to Vibrio cholerae 01, biovar cholerae'), 
 ('A00.1', 'Cholera due to Vibrio cholerae 01, biovar eltor'), 
 ('A00.9', 'Cholera, unspecified')]