The module adds syntactic sugar to Python 3 to allow concise specification of templates. I use it to generate a collection of static web pages from various data sources.
I wanted something relatively simple without the overhead of, say, cheetah or more recent templating systems (e.g. Mako and Jinja2). (I don't want to separate the templating logic from the templates, nor do I need direct integration with a web server. I just want to be able to maintain a collection of static web pages built from an occasionally changing source of data.)
The approach for this module is to extend Python 3 with just a few elements of syntactic sugar that help specify templates in python particularly concisely. But to do that, the module does several hackish things: it hooks into the import system, and it rewrites the abstract syntax trees of imported template files to implement the syntactic sugar.
I know the code is currently rough, error handling is rudimentary. I'm mainly looking for feedback on the overall approach. For example, this approach intentionally sacrifices modularity and the principle of "making things explicit", to obtain convenience and conciseness in specifying templates. Is there a way to obtain those benefits without those sacrifices? I'd also like to know of similar existing modules.
The code is hosted at Github. Here is the package structure:
. ├── __init__ ├── compile ├── gather └── load
The compile
, gather
, and load
files are private, they cannot be imported by the user. The package itself is used solely through import template
or import template.xxx
statements, as described in the doc string in __init.py
and the readme (the readme is more readable). Here is the code:
File __init__.py
#!/usr/bin/env python3
'''Syntactic sugar for convenient template generation in python3.
The "template" module tweaks the python3 import system so that (assuming
the template module is on the python import path) the statement
import template.xxx
will locate a file "xxx.pyt" by searching sys.path, then execute that
file as python but with modified "template" semantics:
1. Before execution, in every string constant, each {{...}} substring is
"dequoted". E.g. "a{{b}}c" is replaced by "a" + str(b) + "c". Dequoting
nests: "a {{f('{{x}} b')}} c" becomes "a " + str(f(str(x) + " b")) + "c".
Also, substrings starting with "##" are removed (until the end of
line; this is for commenting multi-line strings).
2. Each function definition is modified so that during execution of
the function, whenever a statement that consists solely of an
expression is executed, the value of the expression is remembered.
(In normal python it would be discarded.) If the function then has
return value None, the return value is modified to instead be the
concatenation of the remembered values (cast to str's). Example: the
function render defined below would return "1 Render a=A, f()=f A B."
# file child.pyt
import template
a = 'A'; b = 'B'; f = lambda: 'f {{a}} {{b}}'
def render():
1
" Render a={{a}}, f()={{f()}}."
3. All .pyt files imported by "import template.xxx" are executed in
the namespace (that is, globals()) of the module that imports the
template module (both for variables and for function definitions).
Example:
# file parent.pyt
import template.child
a = 'X'; f = lambda: 'F {{a}} {{b}}'
Then render() can be called in parent.pyt, and it would return the
string "1 Render a=X, f()=F X B."
4. If the module that first imports the template module (via any
"import template" or "import template.xx" statement) is __main__ (the
top-level module, the one that is executed initially), and __main__'s
is itself a .pyt file (its name ends in '.pyt'), then __main__ is also
executed using template semantics. In this case, the "import
template.." statements should be at the top of __main__, before any
non-import statement, and __main__ or one of the template files it
imports directly or indirectly should define a "render" function as in
the example above. After __main__ is executed, render() is
automatically called, and its return value is printed to sys.stdout.
Hence, executing "python3 parent.pyt" in the shell would print "1
Render a=X. f()=X B.".
'''
import sys
import os
import inspect
import types
__all__ = ["load"]
# ################################################ module constants
file_extension = '.pyt'
gatherer_function_name = '_template_gather_'
decorator_name = '_template_decorator_'
# details of module that imports this module
_importer_stack_depth = next(i for i, f in enumerate(inspect.stack())
if i > 1 and f[3] == '<module>')
_importer_stack_frame = inspect.stack()[_importer_stack_depth][0]
_importer_globals = _importer_stack_frame.f_globals
_importer_module_name = _importer_globals['__name__']
_importer_module = sys.modules[_importer_module_name]
importer_filename = _importer_globals['__file__']
importer_is_main = _importer_module_name == '__main__'
assert importer_is_main == \
(_importer_stack_depth == len(inspect.stack())-1)
host_module = _importer_module
host_module_globals = _importer_module.__dict__
from .gather import gather, decorator
from .load import loader, exec_template_in_host_module
# ################################################ try_render
def try_render():
global host_module_globals
# https://docs.python.org/2/library/atexit.html
render = host_module_globals.get("render")
if isinstance(render, types.FunctionType):
# print("calling render")
try:
print(render(), end="")
status = 0
except:
import traceback
traceback.print_exc(file=sys.stderr)
status = 1
else:
print("no template render function defined", file=sys.stderr)
status = 1
return status
# ################################################ CODE
host_module_globals[gatherer_function_name] = gather
host_module_globals[decorator_name] = decorator
# set up support of "import template.xxx"
sys.meta_path.append(loader)
# load is global
def load(module_name):
'''
"template.load('x')" is equivalent to "import template.x",
except it also works when the module name contains periods.
'''
loader.load_module(module_name)
# If __main__ imported us and is a pyt file,
# then reload __main__ as pyt template,
# then call render() and exit.
basename, extension = os.path.splitext(importer_filename)
if importer_is_main and extension == file_extension:
exec_template_in_host_module(importer_filename)
sys.exit(try_render())
File compile.py
:
#!/usr/bin/env python3
import ast
import sys
import re
import itertools
from . import decorator_name, gatherer_function_name
_filename = None
def compile_template_file(filename):
'''
Compile template file into a python code object.
'''
with open(filename) as f:
code = f.read()
# Template file syntax is valid Python syntax,
# but with slightly different semantics.
# 1. Parse the template file using Python syntax.
template_AST = ast.parse(code, filename=filename, mode='exec')
# 2. Change the syntax tree to implement the modified semantics.
global _filename, _pyt_to_python
_filename = filename
python_AST = _pyt_to_python.visit(template_AST)
ast.fix_missing_locations(python_AST)
# 3. Compile the modified syntax tree as Python.
return compile(python_AST, filename, mode='exec', dont_inherit=True)
def _split_by_braces(strng):
'''
Split strng into pieces separated by _top_level_ pairs of braces.
e.g. split 'aa{{b{{c}} }}d{{e}}' to ['aa', ' b{{c}} ', 'd', 'e', ''].
For each piece, return (depth, piece). Depth alternates 0 and 1.
Used below to implement the dequote mechanism.
'''
start = depth = 0
for match in re.finditer(r"{{|}}", strng):
prev_depth = depth
depth += 1 if match.group(0) == '{{' else -1
if depth < 0:
raise Exception("unbalanced format string " + strng)
if depth == 0 or prev_depth == 0:
yield strng[start:match.start()]
start = match.end()
if depth > 0:
raise Exception("unbalanced format string " + strng)
yield strng[start:len(strng)]
class _Pyt_to_python(ast.NodeTransformer):
'''
Given pyt abstract syntax tree (AST), transform it into a python AST.
See e.g. https://greentreesnakes.readthedocs.org/en/latest/.
'''
def visit_Str(self, node):
'''
Dequote each string constant (expand {{...}} appropriately).
'''
def str_node(substr):
assert isinstance(substr, str)
return ast.Str(s=substr)
def parse_tree(substr):
global _filename
try:
arg_node = ast.parse("str(" + substr + ")",
filename="<format string>",
mode='eval').body
except:
print("format parse error", file=sys.stderr)
print('substring', substr, file=sys.stderr)
print('File "' + _filename +
'", line', node.lineno, file=sys.stderr)
sys.exit(1)
# recursively process the new subtree
return self.generic_visit(arg_node)
s = node.s
s = re.sub(r"##.*", "", s) # remove comments
if not s:
return str_node("")
# split s around its {{ ... }} segments
# see doc for _split_by_braces
split = tuple(_split_by_braces(s))
if len(split) == 1:
return ast.Str(s=split[0])
args = [f(substr) for (substr, f) in
zip(split, itertools.cycle((str_node, parse_tree)))]
# build and return node for "".join(tuple(args))
func_node = ast.Attribute(value=str_node(""),
attr="join",
ctx=ast.Load())
call_node = ast.Call(func=func_node,
args=[ast.Tuple(elts=args, ctx=ast.Load())],
keywords=[],
starargs=None,
kwargs=None)
return call_node
def visit_Expr(self, node):
'''
Modify the tree to call the gather function on the value of
each Expr in the template. (Expr's are statement-expressions.)
'''
node = self.generic_visit(node)
func = ast.Name(id=gatherer_function_name,
ctx=ast.Load())
newvalue = ast.Call(func=func,
args=[node.value],
keywords=[],
starargs=None,
kwargs=None)
newexpr = ast.Expr(value=newvalue)
return newexpr
def visit_FunctionDef(self, node):
'''
Add the decorator to every function definition.
'''
node = self.generic_visit(node)
func = ast.Name(id=decorator_name, ctx=ast.Load())
node.decorator_list.append(func)
return node
_pyt_to_python = _Pyt_to_python()
File gather.py
#!/usr/bin/env python3
import functools
import sys
active = None
def gather(value):
'''
Append str(value) to currently active list of values.
The template compiler adds a call to this function for
every Expr (statement expression) executed, passing in
the value of the Expr after it is evaluated.
'''
if active is not None and value not in ('', None):
active.append(str(value))
def decorator(fn):
'''
Decorate a function to start a new active list of values
before each execution of the function, and to return
the concatenated list of gathered values when the function
returns (if the function would otherwise return None.
The template compiler adds this decorator to every function
definition in the template.
'''
try:
fn = getattr(fn, '_template_wraps')
except AttributeError:
pass
@functools.wraps(fn)
def fn2(*args, **kwds):
global active
tmp, active = active, []
result1 = fn(*args, **kwds)
result2 = "".join(active)
active = tmp
if result1 is None:
return result2
if result2 != "":
print("template.py warning: discarding gathered value",
'"' + result2 + '"',
"from function", fn.__name__,
file=sys.stderr)
return result1
fn2._template_wraps = fn
return fn2
File load.py
#!/usr/bin/env python -w
import sys
import os
from .compile import compile_template_file
from . import file_extension, host_module, host_module_globals
class _Loader:
def find_module(self, template_module_name, path=None):
''' See Python 3 documentation for sys.meta_path '''
return self if template_module_name.startswith("template.") else None
def load_module(self, template_module_name):
'''
Load template module (as a result of import template.xxx)
'''
assert template_module_name.startswith("template.")
try:
return sys.modules[template_module_name]
except KeyError:
pass
filename = template_module_name[len("template."):] + \
file_extension
for d in sys.path:
file_path = os.path.join(d, filename)
if os.path.isfile(file_path):
break
else:
print("import template, file not found:",
filename, file=sys.stderr)
raise ImportError
sys.modules.setdefault(template_module_name, host_module)
return exec_template_in_host_module(file_path)
loader = _Loader()
def exec_template_in_host_module(filename):
'''
Compile template file and execute it in host module namespace
'''
global host_module_globals, host_module
assert os.path.splitext(filename)[1] == file_extension
# print("INJECTING", filename, file=sys.stderr)
code_obj = compile_template_file(filename)
exec(code_obj, host_module_globals)
return host_module
Here's a previous post involving templating in Python.
_temple_init()
? You know that importing a script will execute all the code in it, right? \$\endgroup\$