Typesafe Unit Code Generation

Question

I've previously posted some code that deals with making Cartesian co-ordinates safe at compile time.

I've been doing a bit more thinking about this in a more general setting, as it is a problem that I quite consistently run into in code bases (all quantities are just declared as double and then forgotten about; or at best given a small comment about what unit it is supposed to be in). Making the unit an explicit part of the type, and using the type system to enforce this, is something I think is of great benefit.

Given that a large number of unit conversions require only multiplication/division by some constant factor, theoretically there should be a lot of similarity between code that deals with making various unit systems type safe.

To this end, I've written something in Python that can generate C++ implementations for any unit types that have simple conversions between each other. First, an example of the usage.

u = UnitGenerator('frequency', 'Hz', ['kHz', 'MHz', 'GHz'])
u.add_conversion('kHz', 1e3)
u.add_conversion('MHz', 1e6)
u.add_conversion('GHz', 1e9)
u.has_plus_operator()
u.has_minus_operator()
u.generate('frequency.hpp')

This will generate the following C++11/14 code:

#ifndef FREQUENCY_AUTOGENERATED_HPP_INCLUDED_
#define FREQUENCY_AUTOGENERATED_HPP_INCLUDED_

#include <string>
#include <functional>

namespace unit
{

template <typename T>
class frequency
{
public:

    constexpr explicit frequency(double v)
        : value_(v)
    { }

    constexpr double value() const
    {
        return value_;
    }

private:

    double value_;
};

struct Hz
{
    static std::string to_string()
    {
        return {"Hz"};
    }

    constexpr static frequency<Hz> from_hz(double v)
    {
        return frequency<Hz>(v);
    }

    constexpr static frequency<Hz> to_hz(double v)
    {
        return frequency<Hz>(v);
    }
};

template <typename T>
struct basic_convert
{
    const static double factor;

    constexpr static frequency<T> from_hz(double v)
    {
        return frequency<T>(v / factor);
    }

    constexpr static frequency<Hz> to_hz(double v)
    {
        return frequency<Hz>(v * factor);
    }

protected:

    ~basic_convert() = default;
};

struct kHz
    : public basic_convert<kHz>
{
    static std::string to_string()
    {
        return {"kHz"};
    }
};

struct MHz
    : public basic_convert<MHz>
{
    static std::string to_string()
    {
        return {"MHz"};
    }
};

struct GHz
    : public basic_convert<GHz>
{
    static std::string to_string()
    {
        return {"GHz"};
    }
};

template <>
const double basic_convert<kHz>::factor = 1000.0;

template <>
const double basic_convert<MHz>::factor = 1000000.0;

template <>
const double basic_convert<GHz>::factor = 1000000000.0;

template <typename T>
std::ostream& operator<<(std::ostream& os, frequency<T> unit)
{
    return os << unit.value() << T::to_string();
}

inline namespace literals
{

frequency<Hz> operator"" _Hz(long double d)
{
    return frequency<Hz>(d);
}

frequency<Hz> operator"" _Hz(unsigned long long int d)
{
    return frequency<Hz>(d);
}

frequency<kHz> operator"" _kHz(long double d)
{
    return frequency<kHz>(d);
}

frequency<kHz> operator"" _kHz(unsigned long long int d)
{
    return frequency<kHz>(d);
}

frequency<MHz> operator"" _MHz(long double d)
{
    return frequency<MHz>(d);
}

frequency<MHz> operator"" _MHz(unsigned long long int d)
{
    return frequency<MHz>(d);
}

frequency<GHz> operator"" _GHz(long double d)
{
    return frequency<GHz>(d);
}

frequency<GHz> operator"" _GHz(unsigned long long int d)
{
    return frequency<GHz>(d);
}

} // end inline namespace literals

using frequency_kHz = frequency<kHz>;
using frequency_MHz = frequency<MHz>;
using frequency_GHz = frequency<GHz>;

template <typename U, typename T>
frequency<U> to(frequency<T> from)
{
    frequency<Hz> as_cannon = T::to_hz(from.value());
    return U::from_hz(as_cannon.value());
}

template <typename U, typename T, typename Operator>
frequency<U> base_operator(frequency<U> v1, frequency<T> v2, Operator op)
{
    auto as_u = to<U>(v2);
    double total = op(v1.value(), as_u.value());
    return frequency<U>(total);
}

template <typename U, typename T>
frequency<U> operator+(frequency<U> v1, frequency<T> v2)
{
    return base_operator(v1, v2, std::plus<double>());
}

template <typename U, typename T>
frequency<U> operator-(frequency<U> v1, frequency<T> v2)
{
    return base_operator(v1, v2, std::minus<double>());
}

template <typename T, typename U, typename Comparison>
bool base_compare(frequency<T> v1, frequency<U> v2, Comparison comp)
{
    auto as_t = to<T>(v2);
    return comp(v1.value(), as_t.value());
}

template <typename T, typename U>
bool operator==(frequency<T> v1, frequency<U> v2)
{
    return base_compare(v1, v2, std::equal_to<double>());
}

template <typename T, typename U>
bool operator>(frequency<T> v1, frequency<U> v2)
{
    return base_compare(v1, v2, std::greater<double>());
}

template <typename T, typename U>
bool operator<(frequency<T> v1, frequency<U> v2)
{
    return base_compare(v1, v2, std::less<double>());
}

template <typename T, typename U>
bool operator!=(frequency<T> v1, frequency<U> v2)
{
    return base_compare(v1, v2, std::not_equal_to<double>());
}

template <typename T, typename U>
bool operator>=(frequency<T> v1, frequency<U> v2)
{
    return base_compare(v1, v2, std::greater_equal<double>());
}

template <typename T, typename U>
bool operator<=(frequency<T> v1, frequency<U> v2)
{
    return base_compare(v1, v2, std::less_equal<double>());
}


} // end namespace unit

#endif // FREQUENCY_AUTOGENERATED_HPP_INCLUDED_

A (really short) example of using some of the code it generates:

#include "frequency.hpp"
#include <iostream>

int main()
{
    using namespace unit::literals;

    auto freq_1 = 50.0_kHz;
    auto freq_2 = 10.1_MHz;

    auto f3 = freq_1 + freq_2;
    std::cout << f3 << '\n';
    auto f4 = freq_2 + freq_1;
    std::cout << f4 << '\n';

    std::cout << std::boolalpha << (freq_1 < freq_2) << '\n';
}

The actual Python code that generates everything:

'''
unit_generator.py
------------------

Code-generator that generates C++ code implementing type-safe units
and unit conversions. This can be used for a wide variety of different
unit types, such as weights, angles, times, frequencies etc.

This makes a few assumptions:

    - The underlying unit value is to be represented by a floating
      point value (specifically, a double).
    - Conversion between each unit type is a matter of applying a
      multiplication or division by a constant factor.

There is also the idea of a "canonical" unit. This is the
user-chosen unit in a set that all other units will be able
to convert to and from. This cuts down on the required number
of conversion functions (2N vs N^2), as every conversion can
be done by converting to and then from the canonical unit.
'''

from collections import OrderedDict
from itertools import chain, repeat
from math import pi
import re
import string
import textwrap

class UnitGenerator:

    def __init__(self, unit_type, canonical, units):
        self._unit_type = unit_type
        # Want to keep units in the same order as what we are given.
        # Python doesn't have an OrderedSet, but an OrderedDict
        # where the values are None suffices.
        self._units = OrderedDict(zip(units, repeat(None)))
        # Cannonical unit of the set. This is an arbitrary 
        #(user-decided) unit that the other units in the set can 
        # be converted from/to.
        self._canonical = canonical
        # Contains the constant factors that shift the given unit
        # type to the canonical unit.
        # E.g. if the canonical unit is kilograms (kg), and one
        # of the other unit types is grams (g), then there will
        # be an entry like {'g': 1e-3}. This corresponds to
        # the amount we need to multiply a gram by to represent
        # it as a kilogram.
        self._factors = {}
        self._namespace = 'unit'
        self._namespace_regex = re.compile('[a-zA-Z][_a-zA-Z0-9]+')
        # Will hold the list of arithmetic operators the user would
        # like to be generated (e.g. operator+).
        self._operators = {}

    def add_unit(self, unit):
        '''Adds another unit to the set of units.''' 
        self._units[unit] = None

    def add_conversion(self, unit, factor):
        '''
        Adds a conversion factor between the canonical
        unit and the unit parameter passed in.
        Note that this should be the factor that converts FROM the
        given unit parameter TO the canonical unit.
        Example:

        Say time was the unit of interest. Let seconds (s)
        be the canonical unit. Let milliseconds (ms) be 
        one of the time units that we want to be able to convert
        from/to. Then the call would be:

            add_conversion('ms', 1000)
        '''
        if unit not in self._units:
            raise NameError('{} is not a valid unit'.format(unit))
        self._factors[unit] = factor

    @property
    def namespace(self):
        '''
        Returns the namespace that all of the generated
        code will live under.
        '''
        return self._namespace

    @namespace.setter
    def namespace(self, new_namespace):
        '''Changes the namespace that the code will live under.
        Note that this namespace name must be valid, and is not
        allowed to start with a number or an underscore.
        '''
        match = self._namespace_regex.match(new_namespace)
        length = len(new_namespace)
        if all((match,  match.start() == 0,  match.end() == length)):
            self._namespace = new_namespace

    def has_plus_operator(self):
        '''Will generate an operator+.'''
        self._operators['+'] = 'std::plus<double>()'

    def has_minus_operator(self):
        '''Will generate an operator-.'''
        self._operators['-'] = 'std::minus<double>()'

    def generate(self, path):
        '''Generates the actual C++ code to the given path.'''
        with open(path, 'w') as output_file:
            output_file.write(self._base_header())
            output_file.write(self._base_class())
            output_file.write(self._canonical_unit())
            output_file.write(self._convert_base_class())
            for unit_struct in self._unit_structs():
                output_file.write(unit_struct)
            for factor in self._factor_code():
                output_file.write(factor)
            output_file.write(self._ostream_operator())
            for literals in self._literal_operators():
                output_file.write(literals)
            for using in self._using_declarations():
                output_file.write(using)
            output_file.write(self._conversion_to())
            for operator_code in self._operator_code():
                output_file.write(operator_code)
            for comparison_code in self._comparison_code():
                output_file.write(comparison_code)
            output_file.write(self._footer())

    def _base_header(self):
        '''
        Returns a string containing the header guard, 
        required includes, and opens the namespace under
        which all the code will live.
        '''

        base_header = string.Template('''
            #ifndef ${unit}_AUTOGENERATED_HPP_INCLUDED_
            #define ${unit}_AUTOGENERATED_HPP_INCLUDED_

            #include <string>
            #include <functional>

            namespace $namespace
            {
        ''')

        header = textwrap.dedent(
            base_header.substitute(
                unit=self._unit_type.upper(),
                namespace=self._namespace
            )
        )

        # Strip the first newline
        return header[1:]

    def _base_class(self):
        '''
        Returns the string containing the base templated
        unit class. This will be given the name of the unit_name
        passed into __init__.
        '''

        base_class_code = string.Template('''
        template <typename T>
        class $unit_name
        {
        public:

            constexpr explicit $unit_name(double v)
                : value_(v)
            { }

            constexpr double value() const
            {
                return value_;
            }

        private:

            double value_;
        };
        ''')

        return textwrap.dedent(
            base_class_code.substitute(
                unit_name=self._unit_type,
                namespace=self._namespace
            )
        )

    def _canonical_unit(self):
        '''
        Returns a string representing the code for the canonical unit.
        This is separate from the rest of the units as the conversion
        code can simply return the value unchanged.
        '''

        unit_code = string.Template('''
        struct $cannon_unit
        {
            static std::string to_string()
            {
                return {"$cannon_unit"};
            }

            constexpr static $unit_class<$cannon_unit> from_${unit_lower}(double v)
            {
                return $unit_class<$cannon_unit>(v);
            }

            constexpr static $unit_class<$cannon_unit> to_${unit_lower}(double v)
            {
                return $unit_class<$cannon_unit>(v);
            }
        };
        ''')

        return textwrap.dedent(
            unit_code.substitute(
                cannon_unit=self._canonical, 
                unit_class=self._unit_type, 
                unit_lower=self._canonical.lower()
            )
        )

    def _convert_base_class(self):
        '''Returns a string with the basic conversion code that
        is specialized by each unit, except the canonical unit.
        '''
        convert_code = string.Template('''
        template <typename T>
        struct basic_convert
        {
            const static double factor;

            constexpr static $unit_class<T> from_${unit_lower}(double v)
            {
                return $unit_class<T>(v / factor);
            }

            constexpr static $unit_class<$cannon_unit> to_${unit_lower}(double v)
            {
                return $unit_class<$cannon_unit>(v * factor);
            }

        protected:

            ~basic_convert() = default;
        };
        ''')

        return textwrap.dedent(
            convert_code.substitute(
                unit_class=self._unit_type,
                cannon_unit=self._canonical,
                unit_lower=self._canonical.lower()
            )
        )

    def _unit_structs(self):
        '''Builds and yields a class for each unit 
        in the list of units.
        This uses the CRTP, and inherits from basic_convert.
        '''

        unit_code = string.Template('''
            struct $unit
                : public basic_convert<$unit>
            {
                static std::string to_string()
                {
                    return {"$unit"};
                }
            };
            ''')

        for unit in self._units.keys():
            yield textwrap.dedent(unit_code.substitute(unit=unit))

    def _factor_code(self):
        '''
        Yields the template specialization of the 
        conversion factor for each unit.
        If a conversion factor has not been given for some
        unit, this will throw a NameError.
        '''

        factor_code = string.Template('''
            template <>
            const double basic_convert<$unit>::factor = $value;
        ''')
        non_cannon = (unit for unit in self._units if unit != self._canonical)
        for unit in non_cannon:
            if unit not in self._factors:
                raise NameError('No conversion factor exists for {}'.format(unit))
            yield textwrap.dedent(
                factor_code.substitute(unit=unit, value=self._factors[unit])
            )

    def _ostream_operator(self):
        '''Returns a string impelementing the ostream<< operator.'''
        ostream_code = string.Template('''
            template <typename T>
            std::ostream& operator<<(std::ostream& os, $unit_name<T> unit)
            {
                return os << unit.value() << T::to_string();
            }
        ''')

        return textwrap.dedent(ostream_code.substitute(unit_name=self._unit_type))

    def _literal_operators(self):
        '''
        Yields strings containing the implementation of the
        user-literal operators for each unit. The suffix used
        is an underscore followed by the unit name.
        All literals are placed in an inner inline namespace
        named "literals".
        '''
        # Generate this for long double and unsigned long long double,
        # so that the the user can use integer values with their
        # literals.
        types = ('long double', 'unsigned long long int')
        literals = string.Template('''
            $unit_name<$unit> operator"" _${unit}($type d)
            {
                return $unit_name<$unit>(d);
            }
        ''')

        yield textwrap.dedent('''
        inline namespace literals
        {
        ''')

        for unit in chain([self._canonical], self._units) :
            for type in types:
                yield textwrap.dedent(
                    literals.substitute(
                        unit_name=self._unit_type, unit=unit, type=type
                    )
                )

        yield textwrap.dedent('''
            } // end inline namespace literals
        ''')

    def _using_declarations(self):
        '''
        Yields strings representing the using declarations.
        This is to partially hide the templates.

        Example:
            If the unit type was time, and the units were
            ['s', 'ms'] corresponding to seconds and milliseconds,
            then this would generate:

            using time_s = time<s>;
            using time_ms = time<ms>;
        '''
        using = string.Template('''
            using ${unit_name}_${unit} = ${unit_name}<$unit>;'''
        )
        for unit in self._units:
            yield textwrap.dedent(
                using.substitute(unit_name=self._unit_type, unit=unit)
            )

    def _conversion_to(self):
        '''
        Returns the implementation of the function "to",
        which can be used to convert between any two types.

        Example:
            If the unit type was time, and the units were
            ['s', 'ms'] corresponding to seconds and milliseconds,
            then the function to could be used to convert to seconds:

                auto as_seconds = to<s>(...)

            or to milliseconds:

                auto as_milliseconds = to<ms>(...)
        '''

        to = string.Template('''

            template <typename U, typename T>
            $unit_name<U> to($unit_name<T> from)
            {
                $unit_name<$cannon_unit> as_cannon = T::to_${cannon_lower}(from.value());
                return U::from_${cannon_lower}(as_cannon.value());
            }
        ''')

        return textwrap.dedent(
            to.substitute(
                unit_name=self._unit_type, 
                cannon_unit=self._canonical,
                cannon_lower=self._canonical.lower()
            )
        )

    def _operator_code(self):
        '''
        Yields the strings containing arithmetic operator code,
        if arithmetic operators have been defined.
        '''
        if not self._operators:
            return

        base_operator = string.Template('''
            template <typename U, typename T, typename Operator>
            $unit<U> base_operator($unit<U> v1, $unit<T> v2, Operator op)
            {
                auto as_u = to<U>(v2);
                double total = op(v1.value(), as_u.value());
                return $unit<U>(total);
            }
        ''')

        yield textwrap.dedent(base_operator.substitute(unit=self._unit_type))

        forward_operator = string.Template('''
            template <typename U, typename T>
            $unit<U> operator$op($unit<U> v1, $unit<T> v2)
            {
                return base_operator(v1, v2, $impl);
            }
        ''')

        for op, impl in self._operators.items():
            yield textwrap.dedent(
                forward_operator.substitute(unit=self._unit_type, op=op, impl=impl)
            )

    def _comparison_code(self):
        '''
        Yields the strings implementing boolean comparisons
        between values.
        '''
        base_compare = string.Template('''
            template <typename T, typename U, typename Comparison>
            bool base_compare($unit<T> v1, $unit<U> v2, Comparison comp)
            {
                auto as_t = to<T>(v2);
                return comp(v1.value(), as_t.value());
            }
        ''')

        yield textwrap.dedent(base_compare.substitute(unit=self._unit_type))

        comp_operator = string.Template('''
            template <typename T, typename U>
            bool operator$op($unit<T> v1, $unit<U> v2)
            {
                return base_compare(v1, v2, $impl);
            }
        ''')

        comparisons = {'<': 'std::less<double>()', 
                       '<=': 'std::less_equal<double>()',
                       '>': 'std::greater<double>()',
                       '>=': 'std::greater_equal<double>()',
                       '==': 'std::equal_to<double>()',
                       '!=': 'std::not_equal_to<double>()'}

        for op, impl in comparisons.items():
            yield textwrap.dedent(
                comp_operator.substitute(unit=self._unit_type, op=op, impl=impl)
            )

    def _footer(self):
        '''Closes the namespace and finishes the include guard.'''
        foot = string.Template('''

            } // end namespace $namespace

            #endif // ${unit}_AUTOGENERATED_HPP_INCLUDED_
        ''')

        return textwrap.dedent(
            foot.substitute(namespace=self._namespace, unit=self._unit_type.upper())
        )

Examples of other unit types that can be generated:

u = UnitGenerator('time', 's', ['ms', 'us', 'ns'])
u.add_conversion('ms', 1e3)
u.add_conversion('us', 1e6)
u.add_conversion('ns', 1e9)
u.has_plus_operator()
u.has_minus_operator()
u.generate('time.hpp')

u = UnitGenerator('angle', 'deg', ['rad'])
u.add_conversion('rad', pi / 180.0)
u.has_plus_operator()
u.has_minus_operator()
u.generate('angle.hpp')

u = UnitGenerator('weight', 'kg', ['pnd', 'g'])
u.add_conversion('pnd', 2.2)
u.add_conversion('g', 1e-3)
u.has_plus_operator()
u.has_minus_operator()
u.generate('weight.hpp')

This isn't complete by any means (missing operators, no comments in generated code, poor naming, plenty of other things...) but it is complete enough that I can throw it up for review. One of the aims is that I want to keep the generated code as readable as possible - if required, it should not be painful to hand-edit it.

Answer 1

Why Code Generation?

This is my main question. Why? What advantage does code generation give you over just writing the class templates in C++ directly? I actually don't think it gives you any - you just end up with this added layer of complexity for where the code comes from.

Furthermore, the generated code isn't particularly easy to follow. We have Hz, kHz, MHz, and GHz. But where those differ is a static constant that is separately defined. That works functionally, but is confusing. Consider an example from <chrono>:

std::chrono::nanoseconds    duration</*signed integer*/, std::nano>
std::chrono::microseconds   duration</*signed integer*/, std::micro>
std::chrono::milliseconds   duration</*signed integer*/, std::milli>
std::chrono::seconds        duration</*signed integer*/>
std::chrono::minutes        duration</*signed integer*/, std::ratio<60>>
std::chrono::hours          duration</*signed integer*/, std::ratio<3600>>

That seems like a much better model. What if we just defined:

using Hz  = frequency<>;
using MHz = frequency<std::mega>;
using GHz = frequency<std::giga>;

There are two things we lose with this approach w.r.t. code generation:

1. Easy naming of types
2. Easy addition of user-defined literals.

However, we'd gain the ability to write:

auto freq = GHz{40};

It's worth mentioning that making the basic_convert destructor private is a very good design choice, since your GHz is just a tag. But I'd rather be able to construct my GHz directly!

Also, it's much easier to write comments in hand-written code than in computer-written code.

Code Duplication

Let's say we went ahead and setup frequency and angle and weight and distance. We now have 4 different function templates for each operator. Now this is all correct, for the sense that your code will not allow me to do stupid things like adding kg and deg. But then... we have four.

Consider a different sort of tagging system:

template <typename tag, typename Value, typename Ratio = std::ratio<1>>
struct Unit { ... };

This would let us define:

struct frequency_tag;
struct angle_tag;
struct weight_tag;

using Hz = Unit<frequency_tag, double>;
using kg = Unit<weight_tag, double, std::kilo>;
using rad = Unit<angle_tag, double, std::ratio<104348,5978700>>; // or write your own ratio class

This lets you write one single operator+ across all of your unit types:

template <typename tag, typename T1, typename R1, typename T2, typename R2>
???? operator+(Unit<tag, T1, R1> lhs, Unit<tag, T2, R2> rhs)
{
    ...
}

Also, this approach sets you down the path of being able to actually implement multiplication of units. How would you do that with your code generation approach?

Your Code Specifically

Minor things I wanted to mention here too. Looking at this from a user perspective:

u = UnitGenerator('frequency', 'Hz', ['kHz', 'MHz', 'GHz'])
u.add_conversion('kHz', 1e3)
u.add_conversion('MHz', 1e6)
u.add_conversion('GHz', 1e9)
u.has_plus_operator()
u.has_minus_operator()

Is it really reasonable to support addition but not subtraction? It may not make sense to have negative units in some cases, but doesn't it always makes sense to subtract when you can add? I'd suggest:

u.has_plusminus_operator()

Also, while you have error checking in case the user forgets to add a conversion (good!), I'd suggest making it more direct. Either all during construction:

u = UnitGenerator('frequency', 'Hz', 
    [('kHz', 1e3), ('MHz', 1e6), ('GHz', 1e9)])

or just all during add_conversion:

u = UnitGenerator('frequency', 'Hz')
u.add_conversion('kHz', 1e3)
u.add_conversion('MHz', 1e6)
u.add_conversion('GHz', 1e9)

That way you don't need error-checking, since there's no way to do it wrong.

One thing you don't error check is the unit name. Should make sure I can't even construct something like:

u = UnitGenerator('1frequency', '3Chainz')

And lastly, consider a fluency model. Have each function return self so that you can write:

(UnitGenerator('time', 's')
.add_conversion('ms', 1e3)
.add_conversion('us', 1e6)
.add_conversion('ns', 1e9)
.has_plusminus_operator()
.generate('time.hpp'))

I don't know how "pythonic" this is - but at least we don't have u!