XML schema parsing and XML creation from flat files

Question

I am new to Python and had to create a schema parser to pull information on attributes and complex types, etc. and then convert data from flat files into the proper XML format. We are processing a lot of data and I've run into some issues once we pass around 1 millions processed records.

I'm looking for any suggestions on code style, best python practices, etc. that will clean this up a bit and increase efficiency.

Here is the part that pulls in the schema:

import csv, ConfigParser, re, datetime, time
from lxml import etree

elemdict = {}
parser = etree.XMLParser()
data = etree.parse(open("testschema.xsd"),parser)
root = data.getroot()
version = root.get("version")

rootelement = root[0]

elements = rootelement[0][0].getchildren()

for e in elements:
    ename = e.get("name")
    elemdict[ename] = [] 
    subelements = e[0][0].getchildren()
    for se in subelements:
        elemdict[ename].append(se.attrib)

specials = root.getchildren()[1:]

specialtypes = {}

for sp in specials:
    sname = sp.get("name") 
    specialtypes[sname] = {}
    specialtypes[sname]["type"] = sp.tag.split('}')[-1] #removes namespace to get either complex or simple type, another option here would be to use xpath('local-name()') or remove the first 34 characters from the tag, none of them really clean options
    typeelements = sp.getchildren()[0].getchildren()
    specialtypes[sname]["details"] = []
    if specialtypes[sname]["type"] == "complexType":
        specialtypes[sname]["requireds"] = []
        for t in typeelements:
            specialtypes[sname]["details"].append(t.get("name"))
            if (not "minOccurs" in t.attrib) or int(t.get("minOccurs"))>0:
                specialtypes[sname]["requireds"].append(t.get("name"))
    else:
        for t in typeelements:
            specialtypes[sname]["details"].append(t.get("value"))

That pulls all information into two dictionaries, elemdict and specialtypes. Special types includes both enums and complex types.

Here is the code that creates the XML elements and returns, where d is the root element, datainput is the line we are obtained from the file, name is the element name, and requireds is a list of required fields for the element:

def addData(d, datainput, name, requireds):

try:
    datavals = dict((k.lower().replace(" ","_"),v) for k,v in datainput.iteritems()) #lower case all the keys for consistency
except AttributeError:
    print "Invalid row, skipping: " + str(datainput)
    return

for i in range(len(requireds)):
    rname = requireds[i]["name"]
    rtype = requireds[i]["type"]
    if rtype[0:3] != "xs:": #if no xs: prefix, element is a complex element of one of our types
        for j in range(len(specialtypes[rtype]["requireds"])):
            spname = specialtypes[rtype]["requireds"][j]
            if not((rname + "_" + spname) in datavals) or len(datavals[(rname + "_" + spname)])<=0:
                print "missing attributes for complex element " + rname + " in values " + str(datavals) + ", returning"
                return
    elif not(rname in datavals) or len(datavals[rname])<=0:
        print "no " + rname + " in values " + str(datavals) + ", returning"
        return


element = etree.SubElement(d, name)
element.set("id", datavals["id"])
for i in range(len(elemdict[name])):
    cname = elemdict[name][i]["name"]
    ctype = elemdict[name][i]["type"]
    if ctype[0:3] != "xs:" and specialtypes[ctype]["type"] == "complexType": #if the type does not start with and xml type and is a complex type specified by us

        validComplex = True
        for j in range(len(specialtypes[ctype]["requireds"])):
            spname = specialtypes[ctype]["requireds"][j]
            if not((cname + "_" + spname) in datavals) or len(datavals[(cname + "_" + spname)])<=0:
                validComplex = False
                break
        if validComplex:
            temp = etree.SubElement(element, cname)
            for d in specialtypes[ctype]["details"]:
                if (cname + "_" + d) in datavals and len(datavals[(cname + "_" + d)]) > 0:
                    tempsubelem = etree.SubElement(temp, d)
                    tempsubelem.text = datavals[(cname + "_" + d)]

    elif cname in datavals and len(datavals[cname]) > 0:
        temp = etree.SubElement(element, cname)
        try:
            if ctype == "xs:date":
                temp.text = str(datetime.date(*time.strptime(datavals[cname],dateformat)[0:3]))
            else:   
                temp.text = datavals[cname]
        except ValueError:
            temp.text = removeNonAscii(datavals[cname])

Answer 1

You might want to rethink your approach in a fundamental way if you want to create millions of rows of XML.

• Define Python classes to contain your actual data. These must be absolutely correct, based on ordinary Python processing. No XSD-based lookup or validation or range checking or anything. Just Python. Your application data must be in Plain Old Python Objects (POPO).

• Write a "serializer" that creates XML from your Plain Old Python Objects. Since the Python objects are already absolutely correct, the output XML will be absolutely correct.

  Or locate a seraializer. http://coder.cl/products/pyxser/ Django has one.

  https://stackoverflow.com/questions/1500575/python-xml-serializers

Once you have this infrastructure of Python classes that produce XML in place, then you do the following.

• Rewrite your XSD parser so that it creates your Plain Old Python Object class definitions from the XSD. It's an XSD->Python translator.

Now you do the bulk of your work in Python. Millions of rows in simple Python class definitions is easy. XML is merely serialized Python objects. Since the XSD is not used, this, too, is easy.

Further (and most importantly) you do a one-time-only conversion of XSD to Python.

Look at this Stackoverflow question for a direction to take. https://stackoverflow.com/questions/1072853/convert-xsd-to-python-class