I'm working on a feature for the HamlPy (Haml for Django) project:
About Haml
For those who don't know, Haml is an indentation-based markup language which compiles to HTML:
%ul#atheletes
- for athelete in athelete_list
%li.athelete{'id': 'athelete_{{ athelete.pk }}'}= athelete.name
compiles to
<ul id='atheletes'>
{% for athelete in athelete_list %}
<li class='athelete' id='athelete_{{ athelete.pk }}'>{{ athelete.name }}</li>
{% endfor %}
</ul>
The code
{'id': 'athelete_{{ athelete.pk }}'}
is referred to as the 'attribute dictionary'. It is an (almost) valid Python dictionary and is currently parsed with some very ugly regular expressions and an eval()
. However, I would like to add some features to it that would no longer make it a valid Python dictionary, e.g. using Haml within the attributes:
%a.link{
'class':
- if forloop.first
link-first
- else
- if forloop.last
link-last
'href':
- url some_view
}
among other things.
I began by writing a class which I could swap out for the eval and would pass all of the current tests:
import re
# Valid characters for dictionary key
re_key = re.compile(r'[a-zA-Z0-9-_]+')
re_nums = re.compile(r'[0-9\.]+')
class AttributeParser:
"""Parses comma-separated HamlPy attribute values"""
def __init__(self, data, terminator):
self.terminator=terminator
self.s = data.lstrip()
# Index of current character being read
self.ptr=1
def consume_whitespace(self, include_newlines=False):
"""Moves the pointer to the next non-whitespace character"""
whitespace = (' ', '\t', '\r', '\n') if include_newlines else (' ', '\t')
while self.ptr<len(self.s) and self.s[self.ptr] in whitespace:
self.ptr+=1
return self.ptr
def consume_end_of_value(self):
# End of value comma or end of string
self.ptr=self.consume_whitespace()
if self.s[self.ptr] != self.terminator:
if self.s[self.ptr] == ',':
self.ptr+=1
else:
raise Exception("Expected comma for end of value (after ...%s), but got '%s' instead" % (self.s[max(self.ptr-10,0):self.ptr], self.s[self.ptr]))
def read_until_unescaped_character(self, closing, pos=0):
"""
Moves the dictionary string starting from position *pos*
until a *closing* character not preceded by a backslash is found.
Returns a tuple containing the string which was read (without any preceding backslashes)
and the number of characters which were read.
"""
initial_pos=pos
while pos<len(self.s):
if self.s[pos]==closing and (pos==initial_pos or self.s[pos-1]!='\\'):
break
pos+=1
return (self.s[initial_pos:pos].replace('\\'+closing,closing), pos-initial_pos+1)
def parse_value(self):
self.ptr=self.consume_whitespace()
# Invalid initial value
val=False
if self.s[self.ptr]==self.terminator:
return val
# String
if self.s[self.ptr] in ("'",'"'):
quote=self.s[self.ptr]
self.ptr += 1
val,characters_read = self.read_until_unescaped_character(quote, pos=self.ptr)
self.ptr += characters_read
# Django variable
elif self.s[self.ptr:self.ptr+2] == '={':
self.ptr+=2
val,characters_read = self.read_until_unescaped_character('}', pos=self.ptr)
self.ptr += characters_read
val="{{ %s }}" % val
# Django tag
elif self.s[self.ptr:self.ptr+2] in ['-{', '#{']:
self.ptr+=2
val,characters_read = self.read_until_unescaped_character('}', pos=self.ptr)
self.ptr += characters_read
val=r"{%% %s %%}" % val
# Boolean Attributes
elif self.s[self.ptr:self.ptr+4] in ['none','None']:
val = None
self.ptr+=4
# Integers and floats
else:
match=re_nums.match(self.s[self.ptr:])
if match:
val = match.group(0)
self.ptr += len(val)
if val is False:
raise Exception("Failed to parse dictionary value beginning at: %s" % self.s[self.ptr:])
self.consume_end_of_value()
return val
class AttributeDictParser(AttributeParser):
"""
Parses a Haml element's attribute dictionary string and
provides a Python dictionary of the element attributes
"""
def __init__(self, s):
AttributeParser.__init__(self, s, '}')
self.dict={}
def parse(self):
while self.ptr<len(self.s)-1:
key = self.__parse_key()
# Tuple/List parsing
self.ptr=self.consume_whitespace()
if self.s[self.ptr] in ('(', '['):
tl_parser = AttributeTupleAndListParser(self.s[self.ptr:])
val = tl_parser.parse()
self.ptr += tl_parser.ptr
self.consume_end_of_value()
else:
val = self.parse_value()
self.dict[key]=val
return self.dict
def __parse_key(self):
'''Parse key variable and consume up to the colon'''
self.ptr=self.consume_whitespace(include_newlines=True)
# Consume opening quote
quote=None
if self.s[self.ptr] in ("'",'"'):
quote = self.s[self.ptr]
self.ptr += 1
# Extract key
if quote:
key,characters_read = self.read_until_unescaped_character(quote, pos=self.ptr)
self.ptr+=characters_read
else:
key_match = re_key.match(self.s[self.ptr:])
if key_match is None:
raise Exception("Invalid key beginning at: %s" % self.s[self.ptr:])
key = key_match.group(0)
self.ptr += len(key)
# Consume colon
ptr=self.consume_whitespace()
if self.s[self.ptr]==':':
self.ptr+=1
else:
raise Exception("Expected colon for end of key (after ...%s), but got '%s' instead" % (self.s[max(self.ptr-10,0):self.ptr], self.s[self.ptr]))
return key
def render_attributes(self):
attributes=[]
for k, v in self.dict.items():
if k != 'id' and k != 'class':
# Boolean attributes
if v==None:
attributes.append( "%s" % (k,))
else:
attributes.append( "%s='%s'" % (k,v))
return ' '.join(attributes)
class AttributeTupleAndListParser(AttributeParser):
def __init__(self, s):
if s[0]=='(':
terminator = ')'
elif s[0]=='[':
terminator = ']'
AttributeParser.__init__(self, s, terminator)
def parse(self):
lst=[]
# Todo: Must be easier way...
val=True
while val != False:
val = self.parse_value()
if val != False:
lst.append(val)
self.ptr +=1
if self.terminator==')':
return tuple(lst)
else:
return lst
The class can be used stand-alone as follows:
>>> from attribute_dict_parser import AttributeDictParser
>>> a=AttributeDictParser("{'id': 'a', 'class': 'b'}")
>>> d=a.parse()
>>> d
{'id': 'a', 'class': 'b'}
>>> type(d)
<type 'dict'>
AttributeDictParser
iterates through characters in s
(the attribute dictionary) and uses the variable ptr
to track its location (to prevent unnecessary string splicing). The function parse_key
parses the keys ('id':
and 'class':
), and the function parse_value
parses the values ('a'
and 'b'
). parse_value works with data types other than strings. It returns False
if it reaches the end of the attribute dictionary, because Null
is a valid value to return.
AttributeTupleAndListParser
parses list and tuple values, as these are valid values (e.g. {'id': ['a','b','c']}
.
Both of these classes inherit from AttributeParser
because they share the same way of parsing values.
Questions:
Is this a sensible approach? Am I insane to think that I can move from
eval()
ing the code as a Python dictionary to a custom parser without causing issues for users, just because it passes the tests?I'm worried that the performance hit of writing a parser in an interpreted language will be too much compared to doing the
eval()
. I've written similar things before for parsing JSON expressions, and was dismayed that for all my optimisations, the two-line regular expression won on the benchmarks. I will do some profiling on it once I've tidied up a few things. Are there any notable ineffeciencies in my approach?There are some things in the old parser that have not been ported to the new one (e.g. supporting the Ruby Haml
=>
syntax). This feature has never been documented however and I doubt anybody knows it's there. What is a good rule of thumb for breaking an undocumented feature in open source projects?I would welcome any feedback on my coding style, as I don't get to be around other developers much.