XML schema parser

Question

I've been working on a lightweight XML schema parser, and have what I think is a moderately clean solution (some parts helped out by previous questions I posted here) so far for obtaining all schema details, but would like any criticism at all that could help further improve this code.

Below I have supplied the schema class I wrote, and then an example schema.txt file that the schema class will open if run as main. The schema class calls under "main" can be modified if you want to get a better look at the schema data structure, and I have some accompanying functions I have written for the class to pull out specific details that I haven't put here because I still need to do some work on them.

schema.py:

from lxml import etree

INDICATORS = ["all", "sequence", "choice"]
TYPES = ["simpleType", "complexType"]

class schema:

    def __init__(self, schemafile):
        if schemafile is None:
            print "Error creating Schema: Invalid schema file used"
            return

        self.schema = self.create_schema(etree.parse(schemafile))

    def create_schema(self, schema_data):
        def getXSVal(element): #removes namespace
            return element.tag.split('}')[-1]

        def get_simple_type(element):
            return {
                "name": element.get("name"),
                "restriction": element.getchildren()[0].attrib,
                "elements": [ e.get("value") for e in element.getchildren()[0].getchildren() ]
        }

        def get_simple_content(element):
            return {
                "simpleContent": {
                    "extension": element.getchildren()[0].attrib,
                    "attributes": [ a.attrib for a in element.getchildren()[0].getchildren() ]
                }
            }

        def get_elements(element):


            if len(element.getchildren()) == 0:
                return element.attrib

            data = {}

            ename = element.get("name")
            tag = getXSVal(element)

            if ename is None:
                if tag == "simpleContent":
                    return get_simple_content(element)
                elif tag in INDICATORS:
                    data["indicator"] = tag
                elif tag in TYPES:
                    data["type"] = tag
                else:
                    data["option"] = tag

            else:
                if tag == "simpleType":
                    return get_simple_type(element)
                else: 
                    data.update(element.attrib)

            data["elements"] = []
            data["attributes"] = []
            children = element.getchildren()        

            for child in children:
                if child.get("name") is not None:
                    data[getXSVal(child)+"s"].append(get_elements(child))
                elif tag in INDICATORS and getXSVal(child) in INDICATORS:
                    data["elements"].append(get_elements(child))
                else:
                    data.update(get_elements(child))

            if len(data["elements"]) == 0:
                del data["elements"]
            if len(data["attributes"]) == 0:
                del data["attributes"]

            return data

        schema = {}
        root = schema_data.getroot()
        children = root.getchildren()
        for child in children:
            c_type = getXSVal(child)
            if child.get("name") is not None and not c_type in schema:
                schema[c_type] = []
            schema[c_type].append(get_elements(child))
        return schema

    def get_Types(self, t_name):
        types = []
        for t in self.schema[t_name]:
            types.append(t["name"])
        return types

    def get_simpleTypes(self):
        return self.get_Types("simpleType")

    def get_complexTypes(self):
        return self.get_Types("complexType")


if __name__ == '__main__':
    fschema = open("schema.txt")

    schema = schema(fschema)

    print schema.get_simpleTypes()
    print schema.get_complexTypes()

schema.txt:

<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" version="3.0" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="main_object">
    <xs:complexType>
      <xs:choice maxOccurs="unbounded">
        <xs:element minOccurs="1" maxOccurs="1" name="source">
          <xs:complexType>
            <xs:all>
              <xs:element name="name" type="xs:string" />
              <xs:element name="group_id" type="xs:integer" />
              <xs:element minOccurs="0" name="description" type="xs:string" />
            </xs:all>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element minOccurs="1" maxOccurs="1" name="event">
          <xs:complexType>
            <xs:all>
              <xs:element name="date" type="xs:date" />
              <xs:element minOccurs="0" name="event_type" type="xs:string" />
              <xs:element minOccurs="0" name="event_hours" type="xs:string" />
            </xs:all>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="state">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="name" type="xs:string" />
            </xs:sequence>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="location">
          <xs:complexType>
            <xs:all>
              <xs:element name="address" type="simpleAddressType" />
              <xs:element minOccurs="0" name="directions" type="xs:string" />
              <xs:element minOccurs="0" name="hours" type="xs:string" />
            </xs:all>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="selection_item">
          <xs:complexType>
            <xs:sequence>
              <xs:element minOccurs="0" name="selection_type_id" type="xs:integer" />
              <xs:element minOccurs="0" maxOccurs="unbounded" name="option_id">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:integer">
                      <xs:attribute name="sort_order" type="xs:integer" />
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="custom_selection">
          <xs:complexType>
            <xs:choice maxOccurs="unbounded">
              <xs:element name="heading" type="xs:string" />
              <xs:sequence maxOccurs="unbounded">
                <xs:element name="response_id">
                  <xs:complexType>
                    <xs:simpleContent>
                      <xs:extension base="xs:integer">
                        <xs:attribute name="sort_order" type="xs:integer" />
                      </xs:extension>
                    </xs:simpleContent>
                  </xs:complexType>
                </xs:element>
              </xs:sequence>
            </xs:choice>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="response">
          <xs:complexType>
            <xs:all>
              <xs:element name="text" type="xs:string" />
              <xs:element minOccurs="0" name="sort_order" type="xs:integer" />
            </xs:all>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
       </xs:choice>
      <xs:attribute fixed="3.0" name="schemaVersion" type="xs:decimal" use="required" />
    </xs:complexType>
  </xs:element>

<xs:complexType name="simpleAddressType">
    <xs:all>
        <xs:element minOccurs="0" name="location_name" type="xs:string"/> 
        <xs:element name="line1" type="xs:string"/>
        <xs:element minOccurs="0" name="line2" type="xs:string"/>
        <xs:element minOccurs="0" name="line3" type="xs:string"/>
        <xs:element name="city" type="xs:string"/>
        <xs:element name="state" type="xs:string"/>
        <xs:element name="zip" type="xs:string"/>
    </xs:all> 
</xs:complexType> 

<xs:complexType name="certified">
  <xs:simpleContent>
    <xs:extension base="xs:integer">
      <xs:attribute name="certification" type="certificationEnum" />
    </xs:extension>
  </xs:simpleContent>
</xs:complexType>

  <xs:simpleType name="certificationEnum">
    <xs:restriction base="xs:string">
      <xs:enumeration value="unofficial_partial"/>
      <xs:enumeration value="unofficial_complete"/>
      <xs:enumeration value="certified"/>
      <xs:enumeration value="Unofficial_partial"/>
      <xs:enumeration value="Unofficial_complete"/>
      <xs:enumeration value="Unofficial_Partial"/>
      <xs:enumeration value="Unofficial_Complete"/>
      <xs:enumeration value="Certified"/>
    </xs:restriction>
  </xs:simpleType> 

  <xs:simpleType name="yesNoEnum">
    <xs:restriction base="xs:string">
      <xs:enumeration value="yes"/>
      <xs:enumeration value="no"/>
      <xs:enumeration value="Yes"/>
      <xs:enumeration value="No"/>
      <xs:enumeration value="YES"/>
      <xs:enumeration value="NO"/>
    </xs:restriction>
  </xs:simpleType> 

</xs:schema>

Answer 1

from lxml import etree

INDICATORS = ["all", "sequence", "choice"]
TYPES = ["simpleType", "complexType"]

class schema:

Python convention is to name classes using CamelCase.

    def __init__(self, schemafile):
        if schemafile is None:
            print "Error creating Schema: Invalid schema file used"
            return

Use exceptions report errors in python. Don't print problems to standard output and then try to continue. Nothing good will come of it. Actually, you don't even need to check for None, because it'll fail on the next line anyways.

        self.schema = self.create_schema(etree.parse(schemafile))

    def create_schema(self, schema_data):
        def getXSVal(element): #removes namespace
            return element.tag.split('}')[-1]

Shouldn't you at least verify that the namespace was correct?

        def get_simple_type(element):
            return {
                "name": element.get("name"),
                "restriction": element.getchildren()[0].attrib,
                "elements": [ e.get("value") for e in element.getchildren()[0].getchildren() ]
        }

It looks like you are using a dictionary like an object. Perhaps you should actually be creating a SimpleType object with these attributes.

        def get_simple_content(element):
            return {
                "simpleContent": {
                    "extension": element.getchildren()[0].attrib,
                    "attributes": [ a.attrib for a in element.getchildren()[0].getchildren() ]
                }
            }

        def get_elements(element):

I've go no idea what this function is trying to do

            if len(element.getchildren()) == 0:
                return element.attrib

            data = {}

            ename = element.get("name")
            tag = getXSVal(element)

            if ename is None:

It seems strange that you check for the name, but don't do anything with it

                if tag == "simpleContent":
                    return get_simple_content(element)

Its confusing the way you sometimes return something, other times you add into a dictionary.

                elif tag in INDICATORS:
                    data["indicator"] = tag
                elif tag in TYPES:
                    data["type"] = tag
                else:
                    data["option"] = tag

            else:
                if tag == "simpleType":
                    return get_simple_type(element)
                else: 
                    data.update(element.attrib)

I don't really follow what the theory for this condition is. I do see the same code showing up multiple times which makes me wonder if it can be refactored to be cleaner.

            data["elements"] = []
            data["attributes"] = []
            children = element.getchildren()        

            for child in children:

Combine the last two lines

                if child.get("name") is not None:
                    data[getXSVal(child)+"s"].append(get_elements(child))
                elif tag in INDICATORS and getXSVal(child) in INDICATORS:
                    data["elements"].append(get_elements(child))
                else:
                    data.update(get_elements(child))

            if len(data["elements"]) == 0:
                del data["elements"]
            if len(data["attributes"]) == 0:
                del data["attributes"]

Do you really want to do this? It seems to me that it'll make code harder to write that uses the data

            return data

These long function as inner functions smell bad. The suggest perhaps they should be in another class or something.

        schema = {}
        root = schema_data.getroot()
        children = root.getchildren()
        for child in children:
            c_type = getXSVal(child)
            if child.get("name") is not None and not c_type in schema:
                schema[c_type] = []

If the name is None, won't that cause the next line to have an error?

            schema[c_type].append(get_elements(child))

Instead use schema.setdefault(c_type,[]).append(get_elements(child)) it'll take care adding the list the first time you append.

        return schema

    def get_Types(self, t_name):

Python convetion is lowercase_with_underscores for method names

        types = []
        for t in self.schema[t_name]:
            types.append(t["name"])
        return types

I'd use return [t["name"] for t in self.schema[t_name]]

    def get_simpleTypes(self):
        return self.get_Types("simpleType")

    def get_complexTypes(self):
        return self.get_Types("complexType")


if __name__ == '__main__':
    fschema = open("schema.txt")

I suggest using with to make sure it gets closed

    schema = schema(fschema)

    print schema.get_simpleTypes()
    print schema.get_complexTypes()

My overall problem with your approach is that you are converting the xml schema into a bunch of unstructured dictionaries. The result isn't going to be much easier to work then the original XML objects. It's gonna be a real pain writing code to work with the schema representation you've produced. Without knowing the overall goal of why you are parsing the schema, I can't really suggest how to improve it.

On your new vesrion

    try:
        self.schema = self.create_schema(etree.parse(schemafile))
    except:
        print "Error creating Schema: Invalid schema file used"

NO! Don't catch the exception. Let the program just die. Never ever ever ever just print to the screen where an error happens.

Here my version of what you've done.

from lxml import etree
from copy import copy
SCHEMA_SPACE = "{http://www.w3.org/2001/XMLSchema}"

class Schema:

    def __init__(self, schemafile):
        self.root = etree.parse(schemafile)

    def findall(self, path):
        return self.root.findall( path.replace("xs:", SCHEMA_SPACE) )

    def find(self, path):
        return self.root.find( path.replace("xs:", SCHEMA_SPACE) )

    def names_of(self, nodes):
        return [node.get("name") for node in nodes]

    def get_Types(self, t_name):
        return self.names_of( self.findall(t_name) ) 

    def get_simpleTypes(self):
        return self.get_Types("xs:simpleType")

    def get_complexTypes(self):
        return self.get_Types("xs:complexType")

    def get_elements_of_attribute(self, attribute):
        return self.names_of(self.findall(".//xs:element/xs:complexType/xs:" + attribute + "/../.."))

    def get_element_attributes(self, name): 

        node = self.find(".//xs:element[@name='" + name + "']")
        if node is None:
            node = self.find(".//xs:complexType[@name='" + name + "']")

        if node is None:
            return None
        else:
            return node.attrib


if __name__ == '__main__':
    with open("schema.txt") as f:

        schema = Schema(f)

        print schema.get_simpleTypes()
        print schema.get_complexTypes()
        print schema.get_elements_of_attribute("all")

        print schema.get_element_attributes("source")
        print schema.get_element_attributes("contact_id")

Copying all the data into python dictionaries is not helping. It is much easier to extract this information directly from the XML.

Answer 2

Well, your code is amazing in that it proves you can handle the most commonly used 10% subset of XML Schema in a few dozen lines of code. But where are you planning to go with this? As you try to increase your coverage of XSD features, your design won't scale. Perhaps that doesn't matter - there's no harm in starting small and being prepared to refactor as it grows.

From my own experience of writing a schema processor, I would recommend keeping a close correspondence between the architecture of your code and the architecture of the spec. That is, implement a data structure corresponding to the schema component model described in the spec; have a process that constructs the SCM from the raw XML representation, and that enforces all the XML representation constraints; have another process that examines the SCM for consistency (enforcing the SCM component constraints), and then another process that's concerned with using the SCM for validation, including generating a finite state automaton or whatever. You can still do it incrementally, of course (e.g leave out named model groups, attribute groups, substitution groups, etc initially) but it's probably a good idea to get the overall structure in place early.

I would follow the XSD 1.1 spec rather than XSD 1.0, partly because the new features are nice for users, but also because the spec is much clearer - still dense, but now penetrable.