3
\$\begingroup\$

For those who don't know, Twine is just a simple interactive fiction making tool. It lets you easily create a series of passages that are hyperlinked to each other, making a choose your own adventure style structure. It exports as HTML format, but if you wanted to just use Twine to write nodes to use elsewhere it's lacking in any other export format. I thought JSON would be a more valuable format to use, so I decide to make this parser.

The source data is a bit of a mess though, here's how it looks:

<tw-storydata name="Sample" startnode="1" creator="Twine" creator-version="2.0.8" ifid="1A382346-FBC1-411F-837E-BAB9EE2FB2E9" format="Harlowe" options=""><style role="stylesheet" id="twine-user-stylesheet" type="text/twine-css"></style><script role="script" id="twine-user-script" type="text/twine-javascript"></script><tw-passagedata pid="1" name="Passage_A" tags="" position="197,62">[[Passage B]]
[[Go to passage C|Passage C]]</tw-passagedata><tw-passagedata pid="2" name="Passage_B" tags="tag-2" position="114,225">This is passage B
[[Passage B]] 
[[Passage A]] </tw-passagedata><tw-passagedata pid="3" name="Passage_C" tags="tag-1 tag-2" position="314,225">This passage goes nowhere.</tw-passagedata></tw-storydata>

In case it's not clear (as it wasn't to me at first), the line breaks are only occuring when the actual text passages contain newline characters. Otherwise all the tags just run on and on, on the same line. This is not at all ideal for parsing, especially if I want to read line by line. So the first step of the process is calling my reformat_html function that will separate tags to one per line and put passages on a line by themselves:

<tw-storydata name="Sample" startnode="1" creator="Twine" creator-version="2.0.8" ifid="1A382346-FBC1-411F-837E-BAB9EE2FB2E9" format="Harlowe" options="">
<style role="stylesheet" id="twine-user-stylesheet" type="text/twine-css">
</style>
<script role="script" id="twine-user-script" type="text/twine-javascript">
</script>
<tw-passagedata pid="1" name="Passage_A" tags="" position="197,62">
[[Passage B]]
[[Go to passage C|Passage C]]
</tw-passagedata>
<tw-passagedata pid="2" name="Passage_B" tags="tag-2" position="114,225">
This is passage B
[[Passage B]] 
[[Passage A]] 
</tw-passagedata>
<tw-passagedata pid="3" name="Passage_C" tags="tag-1 tag-2" position="314,225">
This passage goes nowhere.
</tw-passagedata>
</tw-storydata>

Now I can easily read it line by line, parsing the keyvalue pairs from starting tags, parsing the passage text separate from tags and then knowing when each tag is closed. This tidied up html can now be read into json with my read_as_json function, producing this:

{
    "style": {
        "type": "text/twine-css", 
        "role": "stylesheet", 
        "id": "twine-user-stylesheet"
    }, 
    "script": {
        "type": "text/twine-javascript", 
        "role": "script", 
        "id": "twine-user-script"
    }, 
    "tw-passagedata": [
        {
            "position": "197,62", 
            "text": "[[Passage B]]\n[[Go to passage C|Passage C]]\n", 
            "pid": "1", 
            "name": "Passage_A", 
            "tags": ""
        }, 
        {
            "position": "114,225", 
            "text": "This is passage B\n[[Passage B]] \n[[Passage A]] \n", 
            "pid": "2", 
            "name": "Passage_B", 
            "tags": "tag-2"
        }, 
        {
            "position": "314,225", 
            "text": "This passage goes nowhere.\n\n", 
            "pid": "3", 
            "name": "Passage_C", 
            "tags": "tag-1 tag-2"
        }
    ], 
    "tw-storydata": {
        "startnode": "1", 
        "name": "Sample", 
        "format": "Harlowe", 
        "creator": "Twine", 
        "creator-version": "2.0.8", 
        "ifid": "1A382346-FBC1-411F-837E-BAB9EE2FB2E9", 
        "options": ""
    }
}

Obviously this is a small sample, and I don't do anything to actually parse the passage text at all yet (ie. hyperlinking or formatting), but I'd like to get some feedback on what I've done so far. Some of my parsing feels hacky, but I couldn't think of really elegant ways to check whether characters were inside out outside of quotation marks.

Also I did previously have the <, > and " characters as constants, but the names QUOTETAG and CLOSETAG felt less meaningful, especially when the comments make the context clear regardless.

I'd particularly like to know how readable and accurate this is. I haven't really done parsing before so I might be making some naive mistakes. I also usually don't write code that would need to be used or possibly expanded by other programmers, so making it easy for them is important.

from json import dump
from pprint import pprint


PASSAGE_TAG = "tw-passagedata"
MULTIPLE_TAG_ERROR = "Found multiple '{}' tags, not currently supported"
CLOSETAG_PARSE_ERROR = "Can't parse close tag in {}"


def write_passage(out_file, line):
    """Check how much of line is passage data and write it to out_file

    Returns what remains of the truncated line."""

    end_index = line.find('<')
    if end_index == -1:
        out_file.write(line)
        # Used up all the line as plain passage data.
        return ''
    else:
        # Need a newline so that the tag is separate from the passage data.
        out_file.write(line[:end_index] + '\n')
        return line[end_index:]


def next_quote(line, index):
    """Return the index of the next quote
    Catches a -1 result, not catching this causes infinite loops.
    Add 1 as that's needed for all future searches."""

    quote_index = line[index:].find('"')
    if quote_index == -1
        return 0
    return index + 1 + quote_index


def find_closing_tag(line):
    """Returns the index of the closing tag in line.

    Ensures that it doesn't return a > enclosed in quotes.
    This is because that may just be a character in a string value."""

    close_index = line.find('>')
    quote_index = line.find('"')

    # We need to ensure > isn't enclosed in quotes
    if quote_index != -1:
        # Keep searching until we find a valid closing tag
        while quote_index < close_index:
            quote_index = next_quote(line, quote_index)

            if quote_index > close_index:
                # Find the next > after "
                close_index = (quote_index +
                               line[quote_index:].find('>'))

            # Find the next quote that opens a keyvalue
            quote_index = next_quote(line, quote_index)
            if close_index == -1:
                raise ValueError(CLOSETAG_PARSE_ERROR.format(line))

    return close_index


def reformat_html(filepath):
    """Read Twine2's HTML format and write it out in a tidier format.

    Writes to the same directoy as filepath, just with _temp in the name.
    Returns the filepath of the resulting file."""

    output_file = filepath.replace('.html', '_temp.html')
    with open(filepath) as in_file, open(output_file, 'w') as out_file:
        for line in in_file:
            while line:
                # If it's a passage.
                if not line.startswith('<'):
                    line = write_passage(out_file, line)
                    continue

                close_index = find_closing_tag(line)
                out_file.write(line[:close_index + 1] + '\n')
                line = line[close_index + 1:]

    return output_file


def read_as_json(filepath):
    """Return a dictionary of data from the parsed file at filepath.

    Reads whether a line is a tag, tag closer or text from a passage.
    Close tags are ignored, tag data and passages are parsed into data."""

    data = {}
    with open(filepath) as f:
        for line in f:
            if line.startswith('</'):
                # Closing tag, nothing to see here.
                continue

            if line.startswith('<'):
                # New tag, parse it into data then go to the next line
                parse_tag(line, data)
                continue

            # Anything else is passage data
            # Concatenate it to the current passage node.
            data[PASSAGE_TAG][-1]['text'] += line

    return data


def separate_tags(tag):
    """Takes a tag string and returns the key name and a dictof tag values.

    Tags are strings in the format:
    <tagname key="value" key="another value">

    They're parsed by stripping the <>, then splitting off the tagname.
    Then the rest of the string is read and removed one by one.
    Space and " characters need to be checked to determine whether a space is
    a new keyvalue pair or part of the current value in quotation marks."""

    tagdata = {}
    tag = tag.strip().strip('<>')
    tagname, pairs = tag.split(' ', 1)

    # Makes each loop the same ie always seeking a space character
    pairs += ' '
    while pairs:
        # Find the second quotation mark
        quote_index = pairs.find('"')
        quote_index = 1 + pairs[quote_index + 1:].find('"')

        # If there's no quote found, just find the next space.
        if quote_index == -1:
            space_index = pairs.find(' ')
        # Otherwise find the space after the second quote
        else:
            space_index = quote_index + pairs[quote_index:].find(' ')

        # Add the keyvalue pair that's
        key, value = pairs[:space_index].split('=')
        tagdata[key] = value.strip('"')

        pairs = pairs[space_index + 1:]

    return tagname, tagdata


def parse_tag(tag, data):
    """Parse Twine tag into the data dictionary which is modified in place.

    The tag name is the key, it's value is a dictionary of the tag's key value
    pairs. Passage tags are stored in a list, as of now no other tag should
    be stored this way, and having multiple tags raises a ValueError."""

    tagname, tagdata = separate_tags(tag)
    if tagname == PASSAGE_TAG:
        # Create text string to be available for concatenating to later.
        tagdata['text'] = ''
        try:
            data[tagname].append(tagdata)
        except KeyError:
            data[tagname] = [tagdata]
    else:
        if tagname in data:
            raise ValueError(MULTIPLE_TAG_ERROR.format(tagname))
        data[tagname] = tagdata


if __name__ == "__main__":
    # Sample test
    inpath = r'Sample Data\TwineInput.html'
    outpath = r'Sample Data\FinalOutput.json'
    result = reformat_html(inpath)
    data = read_as_json(result)
    with open(outpath, 'w') as f:
        dump(data, f, indent=4)
\$\endgroup\$
  • \$\begingroup\$ Any particular reason to not use an XML parser (or the BeautifulSoup + lxml combo)? \$\endgroup\$ – 409_Conflict Nov 6 '15 at 13:24
  • \$\begingroup\$ @MathiasEttinger How robust are they? The format of twine data seems erratic, I wasn't sure if it would work with normal xml or html parsing. \$\endgroup\$ – SuperBiasedMan Nov 6 '15 at 13:29
  • 2
    \$\begingroup\$ It seems to me that the data won't be malformed so any parser should do just fine. BeautifulSoup has some comparison between parsers and python's xml.etree.elementtree can even let you access text after a tag if need be. \$\endgroup\$ – 409_Conflict Nov 6 '15 at 13:39
  • \$\begingroup\$ @MathiasEttinger Thanks for the link, I'll look into it. At first glance it seems like it might not honour whitespace but that might only be in the default behaviour. \$\endgroup\$ – SuperBiasedMan Nov 6 '15 at 14:33
2
\$\begingroup\$

Don't reinvent the wheel. You want to parse HTML/XML, use an HTML/XML parser. No matter how tricky the layout seems, as long as well-formed data are fed into them, they should handle it. It’s their job.

Based on your example input, I’ll make the assumption that twine produces well formed XML files. Thus you can get rid of your custom tag splitting/parsing and use the parser of your choice.

For instance, the xml.etree.ElementTree is shipped with the standard library. You can use it to parse your files like:

import xml.etree.ElementTree as ETree

inpath = r'Sample Data\TwineInput.html'
xml = ETree.parse(inpath)
for element in xml.getroot():
    print(element.tag, element.attrib)

which prints:

style {'role': 'stylesheet', 'id': 'twine-user-stylesheet', 'type': 'text/twine-css'}
script {'role': 'script', 'id': 'twine-user-script', 'type': 'text/twine-javascript'}
tw-passagedata {'position': '197,62', 'name': 'Passage_A', 'pid': '1', 'tags': ''}
tw-passagedata {'position': '114,225', 'name': 'Passage_B', 'pid': '2', 'tags': 'tag-2'}
tw-passagedata {'position': '314,225', 'name': 'Passage_C', 'pid': '3', 'tags': 'tag-1 tag-2'}

Pretty close to what you are looking for.

Next thing to do is to take care of multiples tw-passagedata tags, add them a text attribute, handle the case of the root tw-storydata and, possibly, handles duplicates tags with your MULTIPLE_TAG_ERROR message:

import xml.etree.ElementTree as ETree
from json import dump

PASSAGE_TAG = "tw-passagedata"
MULTIPLE_TAG_ERROR = "Found multiple '{}' tags, not currently supported"

def parse_twine_tag(element, data):
    """Parse Twine tag into the data dictionary which is modified in place.

    The tag name is the key, it's value is a dictionary of the tag's key value
    pairs. Passage tags are stored in a list, as of now no other tag should
    be stored this way, and having multiple tags raises a ValueError.
    """

    tagname = element.tag
    attributes = element.attrib

    if tagname == PASSAGE_TAG:
        attributes['text'] = element.text
        data.setdefault(PASSAGE_TAG, []).append(attributes)
    elif tagname in data:
        raise ValueError(MULTIPLE_TAG_ERROR.format(tagname))
    else:
        data[tagname] = attributes

    for child in element:
        parse_twine_tag(child, data)

def parse_twine_file(filepath):
    """Return a dictionary of data from the parsed file at filepath"""

    xml = ETree.parse(filepath)
    data = dict()
    parse_twine_tag(xml.getroot(), data)
    return data

if __name__ == "__main__":
    # Sample test
    inpath = r'Sample Data\TwineInput.html'
    outpath = r'Sample Data\FinalOutput.json'

    data = parse_twine_file(inpath)
    with open(outpath, 'w') as f:
        dump(data, f, indent=4)

outpath, as expected, contains:

{
    "style": {
        "role": "stylesheet", 
        "id": "twine-user-stylesheet", 
        "type": "text/twine-css"
    }, 
    "tw-passagedata": [
        {
            "position": "197,62", 
            "text": "[[Passage B]]\n[[Go to passage C|Passage C]]", 
            "name": "Passage_A", 
            "pid": "1", 
            "tags": ""
        }, 
        {
            "position": "114,225", 
            "text": "This is passage B\n[[Passage B]] \n[[Passage A]] ", 
            "name": "Passage_B", 
            "pid": "2", 
            "tags": "tag-2"
        }, 
        {
            "position": "314,225", 
            "text": "This passage goes nowhere.", 
            "name": "Passage_C", 
            "pid": "3", 
            "tags": "tag-1 tag-2"
        }
    ], 
    "script": {
        "role": "script", 
        "id": "twine-user-script", 
        "type": "text/twine-javascript"
    }, 
    "tw-storydata": {
        "startnode": "1", 
        "name": "Sample", 
        "creator-version": "2.0.8", 
        "ifid": "1A382346-FBC1-411F-837E-BAB9EE2FB2E9", 
        "format": "Harlowe", 
        "options": "", 
        "creator": "Twine"
    }
}
\$\endgroup\$
  • \$\begingroup\$ This is more of a code rewrite/refactor, than a code review. When doing this, I think it is best to clearly state that at the beginning of your answer. \$\endgroup\$ – holroy Nov 6 '15 at 19:14
  • \$\begingroup\$ @holroy like so? Not really sure I added anything valuable this way. \$\endgroup\$ – 409_Conflict Nov 6 '15 at 21:07
  • \$\begingroup\$ @holroy I think it's more a design review. The first code example shows that very clearly IMO. I agree it's a fairly small review, but a review none the less. And personally, design reviews are hard to 'bulk up' with stuff that isn't 'pointless' to the design aspect \$\endgroup\$ – Peilonrayz Nov 6 '15 at 21:13
  • \$\begingroup\$ @holroy I only saw this now, but this fits with my needs for "How do I do this better?" One of my top voted answers was mostly "just use this builtin method", so even if it's different to my approach I'm very happy with this answer. \$\endgroup\$ – SuperBiasedMan Nov 26 '15 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.