Parsing Geographic Data From XML Files in Python

Question

I have recently been working on a program which takes as input, log files generated from a running/biking app. These files are in an xml format which is incredibly regular. In my program, I first strip all of the geographic and time information, and output as a text file. Then, reading back this text file, instances of Entry class are compiled into a list. Using these entries, calculations (and eventually graphing and long term trends) can be achieved.

This python script requires the library geopy to use, and has been tested on Python3.

My main concern is how little of the code is pythonic in nature, with multiple if statements begging the question of optimization. I have uploaded a copy of an example input file for the script on my github, which can be found here. Typing '4-19-17a.bin' without the quotations at the prompt will begin parsing, and will print to standard output the string representation of each Entry instance that was created, along with a couple calculations.

---

dataparse.py

import xml.etree.ElementTree as ET


def processtimestring(time):
    year = int(time[:4])
    month = int(time[5:7])
    day = int(time[8:10])
    h = int(time[11:13])
    m = int(time[14:16])
    s = float(time[17:-6])
    return [year, month, day, h, m, s]

def parsethedata():
    time = []
    lat = []
    lng = []
    alt = []
    dist = []
    garbage = '{http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2}'
    filestring = input('The file to parse: ')
    outputstring = filestring[:-4] + '.txt'
    
    f = open(outputstring, 'w')
    
    tree = ET.parse(filestring)
    root = tree.getroot()
    
    for child in root[0][0][1][4]:
        for info in child:
            if (not info.text):
                for position in info:
                    f.write(position.tag.replace(garbage, '') + '\n')
                    f.write(position.text + '\n')
            else:
                f.write(info.tag.replace(garbage, '') + '\n')
                f.write(info.text + '\n')
    
    f.close()
    
    with open(outputstring) as f:
        for i, line in enumerate(f):
            if ((i+9) % 10 == 0):
                time.append(processtimestring(line[:-1]))
            elif ((i+7) % 10 == 0):
                lat.append(float(line[:-1]))
            elif ((i+5) % 10 == 0):
                lng.append(float(line[:-1]))
            elif ((i+3) % 10 == 0):
                alt.append(float(line[:-1]))
            elif ((i+1) % 10 == 0):
                dist.append(float(line[:-1]))
    return [time, lat, lng, alt, dist]

---

entry.py

import geopy.distance
import datetime as dt


class Entry(object):
    @staticmethod
    def distancecalc(x1, x2):
        coord_1 = (x1.pos[0], x1.pos[1])
        coord_2 = (x2.pos[0], x2.pos[1])
        return geopy.distance.vincenty(coord_1, coord_2).m

    @staticmethod
    def timecalc(x1, x2):
        a, b = x1.time, x2.time
        t1 = dt.datetime(a[0], a[1], a[2], a[3], a[4], int(a[5]))
        t2 = dt.datetime(b[0], b[1], b[2], b[3], b[4], int(b[5]))
        t1_decimal = float(a[5]-int(a[5]))
        t2_decimal = float(b[5]-int(b[5]))
        return (t2 - t1).total_seconds() + (t2_decimal - t1_decimal)

    def __init__(self, time, lat, lng, alt, dist):
        self.time = time
        self.pos = [lat, lng]
        self.alt = alt
        self.dist = dist

    def __str__(self):
        ymd = ('ymd: ' + str(self.time[0]) + ', ' +
               str(self.time[1]) + ', ' + str(self.time[2]))
        hms = (' hms: ' + str(self.time[3]) + ', ' +
               str(self.time[4]) + ', ' + str(self.time[5]))
        lat = ' lat: ' + str(self.pos[0])
        lng = ' lng: ' + str(self.pos[1])
        alt = ' alt: ' + str(self.alt)
        dst = ' dst: ' + str(self.dist)
        stringrepresentation = ymd + hms + lat + lng + alt + dst
        return stringrepresentation

---

main.py

import dataparse as dp
from entry import Entry


exitflag = False

while not exitflag:
    [time, lat, lng, alt, dist] = dp.parsethedata()
    entries = [Entry(x, lat[i], lng[i],
                     alt[i], dist[i]) for i, x in enumerate(time)]
    
    prev = entries[0]
    total = 0.0
    for entry in entries:
        dx = Entry.distancecalc(prev, entry)
        dt = Entry.timecalc(prev, entry)
        total += dx
        print('Total: ' + str(total) + ', Logged Dist: ' + str(entry.dist) +
              ', dx: ' + str(dx) + ', dt: ' + str(dt), end="")
        if dt:
            print(', Speed: ' + str(dx/dt))
        else:
            print(', Speed: 0.0')
        prev = entry
    
    userinput = input('Parse another file? y/n: ')
    
    if userinput == 'n':
        exitflag = True

Answer 1

I have recast your code, and the complete listing is below. I will discuss various elements of the changes here. If I missed explaining a change, or some concept, please leave a comment and I will see what I can do to better cover that.

Organization

First thing I did was move the parsing code to be a method of your Entry class. This parsing code seems to be very specific to this class, so let's make the relationship explicit. I also broke the parsing into parsing a single Entry Node, and the part which manages parsing the entire XML file.

Parsing

File Parse Code:

@classmethod
def from_xml(cls, xml_source):
    tree = etree.parse(xml_source)
    root = tree.getroot()
    return [cls.from_xml_node(child) for child in root[0][0][1][4]]

Some notes:

1. Don't prompt for user input in the parsing code.
2. Convert the data directly to the desired representation (a list of Entries).

Node Parse Code:

converters = dict(
    AltitudeMeters=(float, 'alt'),
    DistanceMeters=(float, 'dist'),
    LatitudeDegrees=(float, 'lat'),
    LongitudeDegrees=(float, 'lng'),
    Time=(dateutil.parser.parse, 'time'),
)

@classmethod
def from_xml_node(cls, node):
    data_point = {}
    for info in node.getiterator():
        tag = info.tag.split('}')[-1]
        if tag in cls.converters:
            converter, tag_text = cls.converters[tag]
            data_point[tag_text] = converter(info.text.strip())
    return cls(**data_point)

A few highlights:

1. Flattened nodes using getiterator

   getiterator finds all of the nodes under a node, so allows the removal of the special if for the nested position data.

2. Removed need for stacked if/else by using converters dict

   By using the node tag as a key into a dict, you can store the pieces needed to parse that tag's value. In this case I stored a conversion function, and the desired name for the value.

3. Removed garbage string by more simply splitting on } and using text after the }

4. Convert the data directly to the desired representation.

   The writing of the data into a secondary file during the XML tree traversal, added quite a bit of complexity, for no apparent gain. Instead this code stores each element of the current node into a dict. The dict is then used to init an Entry object.

5. Dict keys are mapped to same names used in Entry.__init__

   The parameter names for __init__ match the keys used in the data_point dict. This mapping was done via the second element in the tuple in the converters dict. This makes it easy to map things cleanly. This allows return cls(**data_point) to directly create a class instance. Note the fact that you can specify non-keyword parameters with keywords, which is how the **data_point can expand into the 5 parameters to init the Entry class

Class Structure

static methods probably should not be passed a class instance

If you are passing a class instance to a static method of that class, you are probably doing it wrong. Static methods are intended for code which is closely aligned with a class but which it not directly associated with an instance.

So I converted:

@staticmethod
def distancecalc(x1, x2):

To:

def distance_from(self, other):

properties allow calculated fields to be easily treated

In distance_from I converted:

coord_1 = (x1.pos[0], x1.pos[1])
coord_2 = (x2.pos[0], x2.pos[1])
return geopy.distance.vincenty(coord_1, coord_2).m

To:

return geopy.distance.vincenty(self.pos, other.pos).m

By providing:

@property
def pos(self):
    return self.lat, self.lng

Since lat, lng as a pair is a position, making that relationship explicit by using a property, better documents the relationship.

Python features

Using python libraries

By storing the timestamp in the class as a datetime value, I was able to convert:

@staticmethod
def timecalc(x1, x2):
    a, b = x1.time, x2.time
    t1 = dt.datetime(a[0], a[1], a[2], a[3], a[4], int(a[5]))
    t2 = dt.datetime(b[0], b[1], b[2], b[3], b[4], int(b[5]))
    t1_decimal = float(a[5]-int(a[5]))
    t2_decimal = float(b[5]-int(b[5]))
    return (t2 - t1).total_seconds() + (t2_decimal - t1_decimal)

To:

def time_since(self, other):
    return (self.time - other.time).total_seconds()

This removed all of the date conversion and other math.

Use string formatting

By using python string formatting functionality I was able to reduce the __str__ method from 11 lines to 2

def __str__(self):
    return 'time: %s, lat: %.6f, lng: %.6f, alt: %.2f, dst:%.3f' % (
        self.time, self.lat, self.lng, self.alt, self.dist)

zip allows processing multiple lists at once

Since I have completed removed the lists of 'lat', 'lng' etc, this is less relevant, but I thought I would show zip this way:

for prev, entry in zip(entries[:-1], entries[1:]):

this line recasts your print loop from:

prev = entries[0]
for entry in entries:
    ....
    prev = entry

This has the side benefit of not printing the first line of all zeros, but is mostly here to show how to use zip. zip grabs the first element of each of the lists passed to it, and presents them on the first iteration. On the second iteration it presents the second element of each list, etc. In this case I used all but the last element of the entries combined with all but the first element as two lists to get adjacent pairs for each iteration.

Recast Code

import geopy.distance
import dateutil.parser
import xml.etree.ElementTree as etree


class Entry(object):

    def __init__(self, time, lat, lng, alt, dist):
        self.time = time
        self.lat = lat
        self.lng = lng
        self.alt = alt
        self.dist = dist

    def __str__(self):
        return 'time: %s, lat: %.6f, lng: %.6f, alt: %.2f, dst:%.3f' % (
            self.time, self.lat, self.lng, self.alt, self.dist)

    def distance_from(self, other):
        return geopy.distance.vincenty(self.pos, other.pos).m

    def time_since(self, other):
        return (self.time - other.time).total_seconds()

    @property
    def pos(self):
        return self.lat, self.lng

    converters = dict(
        AltitudeMeters=(float, 'alt'),
        DistanceMeters=(float, 'dist'),
        LatitudeDegrees=(float, 'lat'),
        LongitudeDegrees=(float, 'lng'),
        Time=(dateutil.parser.parse, 'time'),
    )

    @classmethod
    def from_xml_node(cls, node):
        data_point = {}
        for info in node.getiterator():
            tag = info.tag.split('}')[-1]
            if tag in cls.converters:
                converter, tag_text = cls.converters[tag]
                data_point[tag_text] = converter(info.text.strip())
        return cls(**data_point)

    @classmethod
    def from_xml(cls, xml_source):
        tree = etree.parse(xml_source)
        root = tree.getroot()
        return [cls.from_xml_node(child) for child in root[0][0][1][4]]


while True:
    filename = input('The file to parse: ')

    entries = Entry.from_xml(filename)

    total = 0.0
    for prev, entry in zip(entries[:-1], entries[1:]):
        dx = entry.distance_from(prev)
        dt = entry.time_since(prev)
        total += dx
        print('Total: %.3f, Logged Dist: %.3f, dx: %.3f, dt: %.3f, '
              'Speed: %.1f' % (total, entry.dist, dx, dt, dx/dt if dt else 0))

    if 'n' == input('Parse another file? y/n: '):
        break