# Parse Twine HTML to JSON

For those who don't know, Twine is just a simple interactive fiction making tool. It lets you easily create a series of passages that are hyperlinked to each other, making a choose your own adventure style structure. It exports as HTML format, but if you wanted to just use Twine to write nodes to use elsewhere it's lacking in any other export format. I thought JSON would be a more valuable format to use, so I decide to make this parser.

The source data is a bit of a mess though, here's how it looks:

<tw-storydata name="Sample" startnode="1" creator="Twine" creator-version="2.0.8" ifid="1A382346-FBC1-411F-837E-BAB9EE2FB2E9" format="Harlowe" options=""><style role="stylesheet" id="twine-user-stylesheet" type="text/twine-css"></style><script role="script" id="twine-user-script" type="text/twine-javascript"></script><tw-passagedata pid="1" name="Passage_A" tags="" position="197,62">[[Passage B]]
[[Go to passage C|Passage C]]</tw-passagedata><tw-passagedata pid="2" name="Passage_B" tags="tag-2" position="114,225">This is passage B
[[Passage B]]
[[Passage A]] </tw-passagedata><tw-passagedata pid="3" name="Passage_C" tags="tag-1 tag-2" position="314,225">This passage goes nowhere.</tw-passagedata></tw-storydata>


In case it's not clear (as it wasn't to me at first), the line breaks are only occuring when the actual text passages contain newline characters. Otherwise all the tags just run on and on, on the same line. This is not at all ideal for parsing, especially if I want to read line by line. So the first step of the process is calling my reformat_html function that will separate tags to one per line and put passages on a line by themselves:

<tw-storydata name="Sample" startnode="1" creator="Twine" creator-version="2.0.8" ifid="1A382346-FBC1-411F-837E-BAB9EE2FB2E9" format="Harlowe" options="">
<style role="stylesheet" id="twine-user-stylesheet" type="text/twine-css">
</style>
<script role="script" id="twine-user-script" type="text/twine-javascript">
</script>
<tw-passagedata pid="1" name="Passage_A" tags="" position="197,62">
[[Passage B]]
[[Go to passage C|Passage C]]
</tw-passagedata>
<tw-passagedata pid="2" name="Passage_B" tags="tag-2" position="114,225">
This is passage B
[[Passage B]]
[[Passage A]]
</tw-passagedata>
<tw-passagedata pid="3" name="Passage_C" tags="tag-1 tag-2" position="314,225">
This passage goes nowhere.
</tw-passagedata>
</tw-storydata>


Now I can easily read it line by line, parsing the keyvalue pairs from starting tags, parsing the passage text separate from tags and then knowing when each tag is closed. This tidied up html can now be read into json with my read_as_json function, producing this:

{
"style": {
"type": "text/twine-css",
"role": "stylesheet",
"id": "twine-user-stylesheet"
},
"script": {
"type": "text/twine-javascript",
"role": "script",
"id": "twine-user-script"
},
"tw-passagedata": [
{
"position": "197,62",
"text": "[[Passage B]]\n[[Go to passage C|Passage C]]\n",
"pid": "1",
"name": "Passage_A",
"tags": ""
},
{
"position": "114,225",
"text": "This is passage B\n[[Passage B]] \n[[Passage A]] \n",
"pid": "2",
"name": "Passage_B",
"tags": "tag-2"
},
{
"position": "314,225",
"text": "This passage goes nowhere.\n\n",
"pid": "3",
"name": "Passage_C",
"tags": "tag-1 tag-2"
}
],
"tw-storydata": {
"startnode": "1",
"name": "Sample",
"format": "Harlowe",
"creator": "Twine",
"creator-version": "2.0.8",
"ifid": "1A382346-FBC1-411F-837E-BAB9EE2FB2E9",
"options": ""
}
}


Obviously this is a small sample, and I don't do anything to actually parse the passage text at all yet (ie. hyperlinking or formatting), but I'd like to get some feedback on what I've done so far. Some of my parsing feels hacky, but I couldn't think of really elegant ways to check whether characters were inside out outside of quotation marks.

Also I did previously have the <, > and " characters as constants, but the names QUOTETAG and CLOSETAG felt less meaningful, especially when the comments make the context clear regardless.

I'd particularly like to know how readable and accurate this is. I haven't really done parsing before so I might be making some naive mistakes. I also usually don't write code that would need to be used or possibly expanded by other programmers, so making it easy for them is important.

from json import dump
from pprint import pprint

PASSAGE_TAG = "tw-passagedata"
MULTIPLE_TAG_ERROR = "Found multiple '{}' tags, not currently supported"
CLOSETAG_PARSE_ERROR = "Can't parse close tag in {}"

def write_passage(out_file, line):
"""Check how much of line is passage data and write it to out_file

Returns what remains of the truncated line."""

end_index = line.find('<')
if end_index == -1:
out_file.write(line)
# Used up all the line as plain passage data.
return ''
else:
# Need a newline so that the tag is separate from the passage data.
out_file.write(line[:end_index] + '\n')
return line[end_index:]

def next_quote(line, index):
"""Return the index of the next quote
Catches a -1 result, not catching this causes infinite loops.
Add 1 as that's needed for all future searches."""

quote_index = line[index:].find('"')
if quote_index == -1
return 0
return index + 1 + quote_index

def find_closing_tag(line):
"""Returns the index of the closing tag in line.

Ensures that it doesn't return a > enclosed in quotes.
This is because that may just be a character in a string value."""

close_index = line.find('>')
quote_index = line.find('"')

# We need to ensure > isn't enclosed in quotes
if quote_index != -1:
# Keep searching until we find a valid closing tag
while quote_index < close_index:
quote_index = next_quote(line, quote_index)

if quote_index > close_index:
# Find the next > after "
close_index = (quote_index +
line[quote_index:].find('>'))

# Find the next quote that opens a keyvalue
quote_index = next_quote(line, quote_index)
if close_index == -1:
raise ValueError(CLOSETAG_PARSE_ERROR.format(line))

return close_index

def reformat_html(filepath):
"""Read Twine2's HTML format and write it out in a tidier format.

Writes to the same directoy as filepath, just with _temp in the name.
Returns the filepath of the resulting file."""

output_file = filepath.replace('.html', '_temp.html')
with open(filepath) as in_file, open(output_file, 'w') as out_file:
for line in in_file:
while line:
# If it's a passage.
if not line.startswith('<'):
line = write_passage(out_file, line)
continue

close_index = find_closing_tag(line)
out_file.write(line[:close_index + 1] + '\n')
line = line[close_index + 1:]

return output_file

"""Return a dictionary of data from the parsed file at filepath.

Reads whether a line is a tag, tag closer or text from a passage.
Close tags are ignored, tag data and passages are parsed into data."""

data = {}
with open(filepath) as f:
for line in f:
if line.startswith('</'):
# Closing tag, nothing to see here.
continue

if line.startswith('<'):
# New tag, parse it into data then go to the next line
parse_tag(line, data)
continue

# Anything else is passage data
# Concatenate it to the current passage node.
data[PASSAGE_TAG][-1]['text'] += line

return data

def separate_tags(tag):
"""Takes a tag string and returns the key name and a dictof tag values.

Tags are strings in the format:
<tagname key="value" key="another value">

They're parsed by stripping the <>, then splitting off the tagname.
Then the rest of the string is read and removed one by one.
Space and " characters need to be checked to determine whether a space is
a new keyvalue pair or part of the current value in quotation marks."""

tagdata = {}
tag = tag.strip().strip('<>')
tagname, pairs = tag.split(' ', 1)

# Makes each loop the same ie always seeking a space character
pairs += ' '
while pairs:
# Find the second quotation mark
quote_index = pairs.find('"')
quote_index = 1 + pairs[quote_index + 1:].find('"')

# If there's no quote found, just find the next space.
if quote_index == -1:
space_index = pairs.find(' ')
# Otherwise find the space after the second quote
else:
space_index = quote_index + pairs[quote_index:].find(' ')

# Add the keyvalue pair that's
key, value = pairs[:space_index].split('=')
tagdata[key] = value.strip('"')

pairs = pairs[space_index + 1:]

return tagname, tagdata

def parse_tag(tag, data):
"""Parse Twine tag into the data dictionary which is modified in place.

The tag name is the key, it's value is a dictionary of the tag's key value
pairs. Passage tags are stored in a list, as of now no other tag should
be stored this way, and having multiple tags raises a ValueError."""

tagname, tagdata = separate_tags(tag)
if tagname == PASSAGE_TAG:
# Create text string to be available for concatenating to later.
tagdata['text'] = ''
try:
data[tagname].append(tagdata)
except KeyError:
data[tagname] = [tagdata]
else:
if tagname in data:
raise ValueError(MULTIPLE_TAG_ERROR.format(tagname))
data[tagname] = tagdata

if __name__ == "__main__":
# Sample test
inpath = r'Sample Data\TwineInput.html'
outpath = r'Sample Data\FinalOutput.json'
result = reformat_html(inpath)
with open(outpath, 'w') as f:
dump(data, f, indent=4)

• Any particular reason to not use an XML parser (or the BeautifulSoup + lxml combo)? – 301_Moved_Permanently Nov 6 '15 at 13:24
• @MathiasEttinger How robust are they? The format of twine data seems erratic, I wasn't sure if it would work with normal xml or html parsing. – SuperBiasedMan Nov 6 '15 at 13:29
• It seems to me that the data won't be malformed so any parser should do just fine. BeautifulSoup has some comparison between parsers and python's xml.etree.elementtree can even let you access text after a tag if need be. – 301_Moved_Permanently Nov 6 '15 at 13:39
• @MathiasEttinger Thanks for the link, I'll look into it. At first glance it seems like it might not honour whitespace but that might only be in the default behaviour. – SuperBiasedMan Nov 6 '15 at 14:33

Don't reinvent the wheel. You want to parse HTML/XML, use an HTML/XML parser. No matter how tricky the layout seems, as long as well-formed data are fed into them, they should handle it. It’s their job.

Based on your example input, I’ll make the assumption that twine produces well formed XML files. Thus you can get rid of your custom tag splitting/parsing and use the parser of your choice.

For instance, the xml.etree.ElementTree is shipped with the standard library. You can use it to parse your files like:

import xml.etree.ElementTree as ETree

inpath = r'Sample Data\TwineInput.html'
xml = ETree.parse(inpath)
for element in xml.getroot():
print(element.tag, element.attrib)


which prints:

style {'role': 'stylesheet', 'id': 'twine-user-stylesheet', 'type': 'text/twine-css'}
script {'role': 'script', 'id': 'twine-user-script', 'type': 'text/twine-javascript'}
tw-passagedata {'position': '197,62', 'name': 'Passage_A', 'pid': '1', 'tags': ''}
tw-passagedata {'position': '114,225', 'name': 'Passage_B', 'pid': '2', 'tags': 'tag-2'}
tw-passagedata {'position': '314,225', 'name': 'Passage_C', 'pid': '3', 'tags': 'tag-1 tag-2'}


Pretty close to what you are looking for.

Next thing to do is to take care of multiples tw-passagedata tags, add them a text attribute, handle the case of the root tw-storydata and, possibly, handles duplicates tags with your MULTIPLE_TAG_ERROR message:

import xml.etree.ElementTree as ETree
from json import dump

PASSAGE_TAG = "tw-passagedata"
MULTIPLE_TAG_ERROR = "Found multiple '{}' tags, not currently supported"

def parse_twine_tag(element, data):
"""Parse Twine tag into the data dictionary which is modified in place.

The tag name is the key, it's value is a dictionary of the tag's key value
pairs. Passage tags are stored in a list, as of now no other tag should
be stored this way, and having multiple tags raises a ValueError.
"""

tagname = element.tag
attributes = element.attrib

if tagname == PASSAGE_TAG:
attributes['text'] = element.text
data.setdefault(PASSAGE_TAG, []).append(attributes)
elif tagname in data:
raise ValueError(MULTIPLE_TAG_ERROR.format(tagname))
else:
data[tagname] = attributes

for child in element:
parse_twine_tag(child, data)

def parse_twine_file(filepath):
"""Return a dictionary of data from the parsed file at filepath"""

xml = ETree.parse(filepath)
data = dict()
parse_twine_tag(xml.getroot(), data)
return data

if __name__ == "__main__":
# Sample test
inpath = r'Sample Data\TwineInput.html'
outpath = r'Sample Data\FinalOutput.json'

data = parse_twine_file(inpath)
with open(outpath, 'w') as f:
dump(data, f, indent=4)


outpath, as expected, contains:

{
"style": {
"role": "stylesheet",
"id": "twine-user-stylesheet",
"type": "text/twine-css"
},
"tw-passagedata": [
{
"position": "197,62",
"text": "[[Passage B]]\n[[Go to passage C|Passage C]]",
"name": "Passage_A",
"pid": "1",
"tags": ""
},
{
"position": "114,225",
"text": "This is passage B\n[[Passage B]] \n[[Passage A]] ",
"name": "Passage_B",
"pid": "2",
"tags": "tag-2"
},
{
"position": "314,225",
"text": "This passage goes nowhere.",
"name": "Passage_C",
"pid": "3",
"tags": "tag-1 tag-2"
}
],
"script": {
"role": "script",
"id": "twine-user-script",
"type": "text/twine-javascript"
},
"tw-storydata": {
"startnode": "1",
"name": "Sample",
"creator-version": "2.0.8",
"ifid": "1A382346-FBC1-411F-837E-BAB9EE2FB2E9",
"format": "Harlowe",
"options": "",
"creator": "Twine"
}
}

• This is more of a code rewrite/refactor, than a code review. When doing this, I think it is best to clearly state that at the beginning of your answer. – holroy Nov 6 '15 at 19:14
• @holroy like so? Not really sure I added anything valuable this way. – 301_Moved_Permanently Nov 6 '15 at 21:07
• @holroy I think it's more a design review. The first code example shows that very clearly IMO. I agree it's a fairly small review, but a review none the less. And personally, design reviews are hard to 'bulk up' with stuff that isn't 'pointless' to the design aspect – Peilonrayz Nov 6 '15 at 21:13
• @holroy I only saw this now, but this fits with my needs for "How do I do this better?" One of my top voted answers was mostly "just use this builtin method", so even if it's different to my approach I'm very happy with this answer. – SuperBiasedMan Nov 26 '15 at 12:42