Simple Python 3 templating module

Question

The module adds syntactic sugar to Python 3 to allow concise specification of templates. I use it to generate a collection of static web pages from various data sources.

I wanted something relatively simple without the overhead of, say, cheetah or more recent templating systems (e.g. Mako and Jinja2). (I don't want to separate the templating logic from the templates, nor do I need direct integration with a web server. I just want to be able to maintain a collection of static web pages built from an occasionally changing source of data.)

The approach for this module is to extend Python 3 with just a few elements of syntactic sugar that help specify templates in python particularly concisely. But to do that, the module does several hackish things: it hooks into the import system, and it rewrites the abstract syntax trees of imported template files to implement the syntactic sugar.

I know the code is currently rough, error handling is rudimentary. I'm mainly looking for feedback on the overall approach. For example, this approach intentionally sacrifices modularity and the principle of "making things explicit", to obtain convenience and conciseness in specifying templates. Is there a way to obtain those benefits without those sacrifices? I'd also like to know of similar existing modules.

The code is hosted at Github. Here is the package structure:

.
├── __init__
├── compile
├── gather
└── load

The compile, gather, and load files are private, they cannot be imported by the user. The package itself is used solely through import template or import template.xxx statements, as described in the doc string in __init.py and the readme (the readme is more readable). Here is the code:

File __init__.py

#!/usr/bin/env python3

'''Syntactic sugar for convenient template generation in python3.

The "template" module tweaks the python3 import system so that (assuming
the template module is on the python import path) the statement

        import template.xxx

will locate a file "xxx.pyt" by searching sys.path, then execute that
file as python but with modified "template" semantics:

1. Before execution, in every string constant, each {{...}} substring is
"dequoted".  E.g. "a{{b}}c" is replaced by "a" + str(b) + "c".  Dequoting
nests: "a {{f('{{x}} b')}} c" becomes "a " + str(f(str(x) + " b")) + "c".

Also, substrings starting with "##" are removed (until the end of
line; this is for commenting multi-line strings).

2. Each function definition is modified so that during execution of
the function, whenever a statement that consists solely of an
expression is executed, the value of the expression is remembered.
(In normal python it would be discarded.)  If the function then has
return value None, the return value is modified to instead be the
concatenation of the remembered values (cast to str's).  Example: the
function render defined below would return "1 Render a=A, f()=f A B."

        # file child.pyt
        import template
        a = 'A'; b = 'B'; f = lambda: 'f {{a}} {{b}}'
        def render():
            1
            " Render a={{a}}, f()={{f()}}."

3. All .pyt files imported by "import template.xxx" are executed in
the namespace (that is, globals()) of the module that imports the
template module (both for variables and for function definitions).
Example:

         # file parent.pyt
         import template.child
         a = 'X'; f = lambda: 'F {{a}} {{b}}'

Then render() can be called in parent.pyt, and it would return the
string "1 Render a=X, f()=F X B."

4. If the module that first imports the template module (via any
"import template" or "import template.xx" statement) is __main__ (the
top-level module, the one that is executed initially), and __main__'s
is itself a .pyt file (its name ends in '.pyt'), then __main__ is also
executed using template semantics.  In this case, the "import
template.."  statements should be at the top of __main__, before any
non-import statement, and __main__ or one of the template files it
imports directly or indirectly should define a "render" function as in
the example above.  After __main__ is executed, render() is
automatically called, and its return value is printed to sys.stdout.
Hence, executing "python3 parent.pyt" in the shell would print "1
Render a=X. f()=X B.".

'''

import sys
import os

import inspect
import types

__all__ = ["load"]

# ################################################ module constants

file_extension = '.pyt'
gatherer_function_name = '_template_gather_'
decorator_name = '_template_decorator_'

# details of module that imports this module
_importer_stack_depth = next(i for i, f in enumerate(inspect.stack())
                             if i > 1 and f[3] == '<module>')
_importer_stack_frame = inspect.stack()[_importer_stack_depth][0]
_importer_globals = _importer_stack_frame.f_globals
_importer_module_name = _importer_globals['__name__']
_importer_module = sys.modules[_importer_module_name]

importer_filename = _importer_globals['__file__']
importer_is_main = _importer_module_name == '__main__'

assert importer_is_main == \
    (_importer_stack_depth == len(inspect.stack())-1)

host_module = _importer_module
host_module_globals = _importer_module.__dict__

from .gather import gather, decorator
from .load import loader, exec_template_in_host_module

# ################################################  try_render


def try_render():
    global host_module_globals

    # https://docs.python.org/2/library/atexit.html
    render = host_module_globals.get("render")
    if isinstance(render, types.FunctionType):
        # print("calling render")
        try:
            print(render(), end="")
            status = 0
        except:
            import traceback
            traceback.print_exc(file=sys.stderr)
            status = 1
    else:
        print("no template render function defined", file=sys.stderr)
        status = 1
    return status

# ################################################  CODE

host_module_globals[gatherer_function_name] = gather
host_module_globals[decorator_name] = decorator

# set up support of "import template.xxx"
sys.meta_path.append(loader)


# load is global

def load(module_name):
    '''
    "template.load('x')" is equivalent to "import template.x",
    except it also works when the module name contains periods.
    '''
    loader.load_module(module_name)

# If __main__ imported us and is a pyt file,
# then reload __main__ as pyt template,
# then call render() and exit.

basename, extension = os.path.splitext(importer_filename)

if importer_is_main and extension == file_extension:

    exec_template_in_host_module(importer_filename)

    sys.exit(try_render())

File compile.py:

#!/usr/bin/env python3

import ast
import sys
import re
import itertools

from . import decorator_name, gatherer_function_name

_filename = None


def compile_template_file(filename):
    '''
    Compile template file into a python code object.
    '''

    with open(filename) as f:
        code = f.read()

    # Template file syntax is valid Python syntax,
    # but with slightly different semantics.
    # 1. Parse the template file using Python syntax.

    template_AST = ast.parse(code, filename=filename, mode='exec')

    # 2. Change the syntax tree to implement the modified semantics.

    global _filename, _pyt_to_python
    _filename = filename
    python_AST = _pyt_to_python.visit(template_AST)

    ast.fix_missing_locations(python_AST)

    # 3. Compile the modified syntax tree as Python.

    return compile(python_AST, filename, mode='exec', dont_inherit=True)


def _split_by_braces(strng):
    '''
    Split strng into pieces separated by _top_level_ pairs of braces.
    e.g. split 'aa{{b{{c}} }}d{{e}}' to ['aa', ' b{{c}} ', 'd', 'e', ''].
    For each piece, return (depth, piece). Depth alternates 0 and 1.

    Used below to implement the dequote mechanism.
    '''
    start = depth = 0
    for match in re.finditer(r"{{|}}", strng):
        prev_depth = depth
        depth += 1 if match.group(0) == '{{' else -1
        if depth < 0:
            raise Exception("unbalanced format string " + strng)
        if depth == 0 or prev_depth == 0:
            yield strng[start:match.start()]
            start = match.end()
    if depth > 0:
        raise Exception("unbalanced format string " + strng)
    yield strng[start:len(strng)]


class _Pyt_to_python(ast.NodeTransformer):
    '''
    Given pyt abstract syntax tree (AST), transform it into a python AST.
    See e.g. https://greentreesnakes.readthedocs.org/en/latest/.
    '''

    def visit_Str(self, node):
        '''
        Dequote each string constant (expand {{...}} appropriately).
        '''

        def str_node(substr):
            assert isinstance(substr, str)
            return ast.Str(s=substr)

        def parse_tree(substr):
            global _filename
            try:
                arg_node = ast.parse("str(" + substr + ")",
                                     filename="<format string>",
                                     mode='eval').body
            except:
                print("format parse error", file=sys.stderr)
                print('substring', substr, file=sys.stderr)
                print('File "' + _filename +
                      '", line', node.lineno, file=sys.stderr)
                sys.exit(1)

            # recursively process the new subtree
            return self.generic_visit(arg_node)

        s = node.s
        s = re.sub(r"##.*", "", s)  # remove comments
        if not s:
            return str_node("")

        # split s around its {{ ... }} segments
        # see doc for _split_by_braces
        split = tuple(_split_by_braces(s))

        if len(split) == 1:
            return ast.Str(s=split[0])

        args = [f(substr) for (substr, f) in
                zip(split, itertools.cycle((str_node, parse_tree)))]

        # build and return node for "".join(tuple(args))
        func_node = ast.Attribute(value=str_node(""),
                                  attr="join",
                                  ctx=ast.Load())
        call_node = ast.Call(func=func_node,
                             args=[ast.Tuple(elts=args, ctx=ast.Load())],
                             keywords=[],
                             starargs=None,
                             kwargs=None)
        return call_node

    def visit_Expr(self, node):
        '''
        Modify the tree to call the gather function on the value of
        each Expr in the template.  (Expr's are statement-expressions.)
        '''
        node = self.generic_visit(node)
        func = ast.Name(id=gatherer_function_name,
                        ctx=ast.Load())
        newvalue = ast.Call(func=func,
                            args=[node.value],
                            keywords=[],
                            starargs=None,
                            kwargs=None)
        newexpr = ast.Expr(value=newvalue)
        return newexpr

    def visit_FunctionDef(self, node):
        '''
        Add the decorator to every function definition.
        '''
        node = self.generic_visit(node)
        func = ast.Name(id=decorator_name, ctx=ast.Load())
        node.decorator_list.append(func)
        return node

_pyt_to_python = _Pyt_to_python()

File gather.py

#!/usr/bin/env python3

import functools
import sys

active = None


def gather(value):
    '''
    Append str(value) to currently active list of values.

    The template compiler adds a call to this function for
    every Expr (statement expression) executed, passing in
    the value of the Expr after it is evaluated.
    '''
    if active is not None and value not in ('', None):
        active.append(str(value))


def decorator(fn):
    '''
    Decorate a function to start a new active list of values
    before each execution of the function, and to return
    the concatenated list of gathered values when the function
    returns (if the function would otherwise return None.

    The template compiler adds this decorator to every function
    definition in the template.
    '''
    try:
        fn = getattr(fn, '_template_wraps')
    except AttributeError:
        pass

    @functools.wraps(fn)
    def fn2(*args, **kwds):
        global active

        tmp, active = active, []
        result1 = fn(*args, **kwds)
        result2 = "".join(active)
        active = tmp

        if result1 is None:
            return result2
        if result2 != "":
            print("template.py warning: discarding gathered value",
                  '"' + result2 + '"',
                  "from function", fn.__name__,
                  file=sys.stderr)
        return result1

    fn2._template_wraps = fn
    return fn2

File load.py

#!/usr/bin/env python -w

import sys
import os

from .compile import compile_template_file
from . import file_extension, host_module, host_module_globals


class _Loader:
    def find_module(self, template_module_name, path=None):
        ''' See Python 3 documentation for sys.meta_path '''
        return self if template_module_name.startswith("template.") else None

    def load_module(self, template_module_name):
        '''
        Load template module (as a result of import template.xxx)
        '''
        assert template_module_name.startswith("template.")
        try:
            return sys.modules[template_module_name]
        except KeyError:
            pass

        filename = template_module_name[len("template."):] + \
            file_extension

        for d in sys.path:
            file_path = os.path.join(d, filename)
            if os.path.isfile(file_path):
                break
        else:
            print("import template, file not found:",
                  filename, file=sys.stderr)
            raise ImportError

        sys.modules.setdefault(template_module_name, host_module)

        return exec_template_in_host_module(file_path)

loader = _Loader()


def exec_template_in_host_module(filename):
    '''
    Compile template file and execute it in host module namespace
    '''
    global host_module_globals, host_module

    assert os.path.splitext(filename)[1] == file_extension

    # print("INJECTING", filename, file=sys.stderr)
    code_obj = compile_template_file(filename)
    exec(code_obj, host_module_globals)

    return host_module

Here's a previous post involving templating in Python.

Answer 1

There's a lot of code here, so I'm just going to look at the documentation, the _split_by_braces function, and the visit_Str method. You'll see that there's more than enough for one answer. Maybe some of the other users here will comment on the other parts of your program.

1. Documentation

1. Most submissions to Code Review have no documentation at all (requiring us to reverse-engineer everything from code), so to see comprehensive documentation like this is excellent.

2. However, it could be improved. What you have here is reference documention: organized by features. But programmers also need user documentation, organized by use cases. A programmer starts by approaching a piece of software with a task: "how do I turn a list of objects into an HTML table?" and only later do they start asking questions about the exact behaviour of comments in nested templates. And most people learn more quickly from examples than from specifications.

   So I would recommend starting the documentation with examples and use cases. Take a look at the Jinja2 and Django Template Language introductions: they both start with examples.

3. The examples are not very motivating. It is hard to see why I would want to output a string like:

   1 Render a=X, f()=F X B.
   

   Good examples make it easier to understand how a software system works, and just as importantly, why someone would want to use it. It's hard to come up with examples that are clear and succinct, but worth the effort. You can see that the Django docs do a slightly better job of this: their first example is a news aggregator (specific and practical), whereas Jinja2's first example is a generic "My Webpage" with navigation.

4. The examples are not automatically checkable: how would you know if they no longer worked? Changing them into doctests would solve this.

2. _split_by_braces

1. There's an example in the docstring that could be turned into a doctest so that it can be automatically checked.

2. If you do this, then you'll see that the example has a mistake:

   Failed example:
       list(_split_by_braces('aa{{b{{c}} }}d{{e}}'))
   Expected:
       ['aa', ' b{{c}} ', 'd', 'e', '']
   Got:
       ['aa', 'b{{c}} ', 'd', 'e', '']
   

3. There is no need to misspell "string" as "strng" to avoid shadowing the string module, since you don't use that module in this code.

4. The docstring says:

   For each piece, return (depth, piece).

   but this is not right (presumably this was true in an earlier version of the code, but then you realised that if depth always alternates then it's superfluous to return it). It's worth getting into the habit of checking the docstring every time you make a change to a function (or better still, change the documentation first).

5. Since you're reusing the same regular expression each time the function is called, consider compiling it just once.

6. It's a bad idea to raise a plain Exception for specific errors (like unbalanced format strings). How can a caller catch just this specific error and no others? Exception is the root of the built-in exception hierarchy, so if a caller tries to catch it, they catch all built-in, non-system-exiting exceptions too.

   It is better to define a specific exception class:

   class UnbalancedError(Exception):
       pass
   

7. When reporting an error involving an invalid string, it's best to give the repr of the string (as the string itself might include whitespace or newlines, which would be confusing). Python's built-in error messages look like this:

   >>> int('abc')
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   ValueError: invalid literal for int() with base 10: 'abc'
   

   so you could use something like this:

   raise UnbalancedError("too many close braces: {!r}".format(string))
   

8. The final error condition is:

   if depth > 0:
   

   (because you are confident that depth < 0 would have been caught earlier) but I would write:

   if depth != 0:
   

   to make it clear that the expected final condition is depth == 0.

Revised code:

class UnbalancedError(Exception):
    pass

_BRACE_PAIR_RE = re.compile("{{|}}")

def _split_by_braces(string):
    '''Generator that splits string into pieces separated by top level
    pairs of braces, and yields the pieces.

    >>> list(_split_by_braces('aa{{ b{{c}} }}d{{e}}'))
    ['aa', ' b{{c}} ', 'd', 'e', '']

    The pieces alternate between unbraced and braced, with the first
    and last pieces unbraced. This may require some of the pieces to
    be empty strings to maintain the alternation:

    >>> list(_split_by_braces('{{a}}{{b}}{{c}}'))
    ['', 'a', '', 'b', '', 'c', '']

    Raises UnbalancedError if the pairs of braces are not properly
    nested:

    >>> list(_split_by_braces('{{{{}}'))
    ... # doctest: +IGNORE_EXCEPTION_DETAIL
    Traceback (most recent call last):
      ...
    UnbalancedError: too many open braces: '{{{{}}'

    '''
    start = depth = 0
    for match in _BRACE_PAIR_RE.finditer(string):
        prev_depth = depth
        depth += 1 if match.group(0) == '{{' else -1
        if depth < 0:
            raise UnbalancedError("too many close braces: {!r}".format(string))
        if depth == 0 or prev_depth == 0:
            yield string[start:match.start()]
            start = match.end()
    if depth != 0:
        raise UnbalancedError("too many open braces: {!r}".format(string))
    yield string[start:len(string)]

3. _Pyt_to_python.visit_Str

1. There are no test cases. Syntax tree transformation code like this is easily broken, so without unit tests it is difficult to reliably make changes to the code. The kind of thing you need is just a long list of templates and their expected output, covering all the awkward cases that you can think of:

   import unittest
   
   class TestTransform(unittest.TestCase):
       _CASES = [
           ('{{}}', ''),
           ('{{    }}', ''),
           ('{{ # comment }}', ''),
           ('{{}}{{}}{{}}', ''),
           ('{{ }}{{ }}{{ }}', ''),
           (' {{ }} {{}} {{}} ', '    '),
           ('{{a}}', '1'),
           ('{{   a    }}', '1'),
           ('{{ a # comment }}', '1'),
           ('{{ # comment\n a }}', '1'),
           ('{{a}}{{b}}{{c}}', '123'),
           ('-{{a}}={{b}}+{{c}}*', '-1=2+3*'),
           ('{{a+b+c}}{{a*b*c}}', '66'),
           ('{{"{{a}}" * 4}}', '1111'),
           # ... and so on ...
       ]
   
       def test_transform(self):
           for template, expected in self._CASES:
               tree = ast.parse(repr(template), '<string>', 'eval')
               code = compile(PytTransformer().visit(tree), '<string>', 'eval')
               found = eval(code, dict(a=1, b=2, c=3))
               self.assertEqual(expected, found)
   

2. This code makes heavy use of node constructors in the ast module, and could be shortened by importing their names:

   from ast import Attribute, Call, Expr, Load, Name, Str, Tuple
   

   Similarly, it could be shortened by taking advantage of the positional and default arguments to these constructors. So instead of:

   func_node = ast.Attribute(value=str_node(""),
                             attr="join",
                             ctx=ast.Load())
   call_node = ast.Call(func=func_node,
                        args=[ast.Tuple(elts=args, ctx=ast.Load())],
                        keywords=[],
                        starargs=None,
                        kwargs=None)
   

   write:

   call_node = Call(Attribute(Str(''), 'join', Load()),
                    [Tuple(args, Load())], [])
   

3. The code uses the global variable _filename to pass the filename from compile_template_file to the transformer class. This is a fragile mechanism (a change to either bit of code would break it) and it prevents the code from being used in multi-threaded programs.

   It would be better to store the filename in the transformer object. This would require the creation of a new transformer object for each call to compile_template_file, thus removing the need for the global variable _pyt_to_python.

4. The helper function str_node isn't needed: you can just use ast.Str.

5. parse_tree uses string concatenation to avoid constructing an extra Call node:

   arg_node = ast.parse("str(" + s + ")", ...)
   

   but this does not work if s contains a comment:

   >>> _Pyt_to_python().visit(ast.parse('"{{# this is a comment}}"'))
   Traceback (most recent call last):
     ...
     File "<format string>", line 1
       str(# this is a comment)
                               ^
   SyntaxError: unexpected EOF while parsing
   

   There is no shortcut for explicitly constructing the Call node!

6. I am not convinced by the need for a special comment mechanism. Once the bug noted above is fixed, someone who wants to include comments in a template string can just use ordinary Python commments inside pairs of braces:

   '... {{# this is a comment}} ...'
   

7. A bare except: is a bad idea, because it catches all exceptions, including KeyboardInterrupt, meaning that the program might incorrectly report a parse error if the user types control-C while it is parsing a template.

8. The code exits the program if it encounters any error (not just a syntax error) in a template string. This seems like a bad idea to me: what if the caller needs to catch the error and continue?

9. What the error-handling code is trying to do is to generate an error with the correct filename and line number for a recursive parse error. You can pass in the correct filename to the ast.parse call, but unfortunately it doesn't take a starting line number, so you have to adjust the line number of the exception object for the syntax error:

   try:
       arg_node = ast.parse("str(" + s + ")",
                            filename=self.filename,
                            mode='eval').body
   except SyntaxError as e:
       e.lineno += node.lineno - 1
       raise
   

10. The generated abstract syntax tree nodes don't have the correct line numbers. These need to be fixed up, using ast.copy_location in the case of newly created nodes, and ast.increment_lineno in the case of nodes resulting from parsing template strings.

11. The transformed parse tree contains empty strings resulting from the alternating output convention of _split_by_braces. These could be discarded.

12. If there are pairs of braces with nothing inside them (or only whitespace and comments), then this results in unnecessary calls to str with no arguments. These could be discarded.

13. The removal of empty strings can result in an empty list, or a list containing a single element. In these cases the join could be avoided.

14. The construction of args using a list comprehension places limitations on what you can do for each item. In particular, it makes it hard to discard empty parts. Sometimes it's simplest to write a plain loop.

Revised code:

def __init__(self, filename='<unknown>'):
    self.filename = filename

# Match Python expression, discarding initial whitespace and comments.
_EXPRESSION_RE = re.compile(r'[ \t]*(?:(?:#.*)?\n[ \t]*)*([^ \t\n#])')

def visit_Str(self, node):
    # Split template string into parts. Alternating parts are:
    # (i) literal strings, wrapped by Str, if non-empty; and
    # (ii) Python expressions, recursively parsed, if not whitespace.
    parts = []
    for s, literal in zip(_split_by_braces(node.s), cycle((True, False))):
        if literal:
            if s:
                parts.append(copy_location(Str(s), node))
        else:
            m = self._EXPRESSION_RE.match(s)
            if m:
                try:
                    tree = ast.parse(s[m.start(1):], self.filename,
                                     'eval').body
                except SyntaxError as e:
                    e.lineno += node.lineno - 1
                    raise
                increment_lineno(tree, node.lineno)
                func = copy_location(Name('str', Load()), node)
                call = Call(func, [self.generic_visit(tree)], [])
                parts.append(copy_location(call, node))

    # Build syntax tree for expression joining the parts together.
    if len(parts) == 1:
        return parts[0]
    empty = copy_location(Str(''), node)
    if parts:
        func = copy_location(Attribute(empty, 'join', Load()), node)
        parts = copy_location(Tuple(parts, Load()), node)
        return copy_location(Call(func, [parts], []), node)
    else:
        return empty