LazyEnum with validation

Question

Motivation

In the standard Python library enum, a subclass of Enum would create all its members upon class creation. This can be a huge waste of time and memory, especially if the class contains lots of members but only a fraction of them would ever get called.

As a practice project on metaclasses, I decided to implement a lazy version of Enum. This is not intended to be a full-fledged library, so I didn't implement most of the features in the standard enum library, just some basic functionalities.

In LazyEnum, a member would be created only when we ask to and it would be created only once. For example, let say MyClass is a subclass of LazyEnum. The first time we call MyClass(1) an object of type MyClass is created. Any subsequent call to MyClass(1) simply return the same object. Moreover, I want to put some validation on member creation, say we may only allow n to be a positive integer when calling MyClass(n).

This idea is inspired by Ethan Furman's reply to my previous question here and also a discussion on Udemy with Fred Baptiste, who is the instructor of the Python 3 Deep Dive series.

Before looking at the code, let me provide an example of how to use LazyEnum.

Example

from lazyenum import LazyEnum

class Product(LazyEnum):
    def _validate_identifier_value(product_id):
        # special method used by metaclass for validation
        return isinstance(product_id, int) and (1001 <= product_id <= 9999)

    COMPANY_NAME = 'Our Example Company'

    def __init__(self, product_id, product_title):
        # no need to store 'product_id' as instance attribute
        self.product_title = product_title

    def __repr__(self):
        return f'Product({self.product_id!r}, {self.product_title!r})'

Remark:

1. LazyEnum is created by a private metaclass _LazyEnumMeta.
2. The first non-self parameter of __init__ is automatically grabbed by the metaclass and cached, so we don't need to set it as an instance attribute. If we write self.product_id = product_id, it would raise an error when we try to initialize a new member. This parameter is called the identifier field and its value is called the identifier value, which uniquely identify each member.
3. The metaclass would look for a method named _validate_identifier_value and use it for validation. It can be defined as a static method or a class method, but if we define it as a class method, we would need to decorate it with @classmethod. Here we just define it as a static method.

Let us see some example outputs. First, we can initialize a member as usual and call it by its identifier value.

>>> prod1 = Product(1001, 'Our Nice First Product')
>>> Product(1001)
Product(1001, 'Our Nice First Product')
>>> prod1 is Product(1001)
True

We can get the identifier value by using .identifier_field or directly calling the instance attribute (.product_id in this case). The .identifier_field would give us a nametuple called Identifier, whose first entry is the attribute name and the second entry is the value.

>>> prod1.identifier_field
Identifier(field_name='product_id', value=1001)
>>> prod1.product_id
1001

Error would be raised if we attempt to create a new member with an existing identifier value. Of course, the same thing would happen if we use an invalid identifier value.

>>> Product(1001, 'This Is Still The First Product')
ValueError: Member with identifier value 1001 already exists. Cannont pass additional arguments ('This Is Still The First Product',) or {}.
>>> Product(1, 'Product With Invaild ID')
ValueError: Identifier field 'product_id' has invalid value 1.

In the regular Enum, you can set aliases to a member. Right now we didn't set any alias, but we can do so using dot notation and see all aliases of a member using .all_aliases. We can also simultaneously create a new member and set an alias to it.

>>> prod1.all_aliases
[]
>>> Product.product_one = prod1
>>> Product.first_product = Product.product_one
>>> prod1.all_aliases
['product_one', 'first_product']
>>> 
>>> Product.product_two = Product(1002, 'The Amazing Second Product')
>>> Product.product_two
Product(1002, 'The Amazing Second Product')
>>> Product(1002).all_aliases
['product_two']

But be careful, we may accidentally overwrite other class attributes.

>>> Product.COMPANY_NAME
'Our Example Company'
>>> Product.COMPANY_NAME = prod1
>>> prod1.all_aliases
['product_one', 'first_product', 'COMPANY_NAME']
>>> Product.COMPANY_NAME
Product(1001, 'Our Nice First Product')
>>> 
>>> Product.COMPANY_NAME = 'Our Example Company'
>>> prod1.all_aliases
['product_one', 'first_product']

We can change instance attributes that are not the identifier field. Attempting to change the identifier field would raise an error.

>>> prod1.product_title = 'First Product With New Name'
>>> prod1
Product(1001, 'First Product With New Name')
>>> prod1.product_id = 2001
AttributeError: can't set attribute

We can iterate over the class members.

>>> Product(1003, 'Even More Amazing Third Product')
Product(1003, 'Even More Amazing Third Product')
>>> for prod in Product: print(prod)
Product(1001, 'First Product With New Name')
Product(1002, 'The Amazing Second Product')
Product(1003, 'Even More Amazing Third Product')
>>> len(Product)
3

Finally, the class has properties .identifier_value_map and .alias_to_member_map, which help inspect all members. Note that we didn't set any alias to Product(1003).

>>> Product.identifier_value_map
mappingproxy({
    1001: Product(1001, 'First Product With New Name'),
    1002: Product(1002, 'The Amazing Second Product'),
    1003: Product(1003, 'Even More Amazing Third Product')
})
>>> Product.alias_to_member_map
mappingproxy(OrderedDict([
    ('product_one', Product(1001, 'First Product With New Name')),
    ('first_product', Product(1001, 'First Product With New Name')),
    ('product_two', Product(1002, 'The Amazing Second Product'))
]))

The Code

Here is the code.

# lazyenum.py
from collections import namedtuple, OrderedDict
from types import MappingProxyType

_Identifier = namedtuple('Identifier', 'field_name value')

def _get_identifier_value(self):
    # use this function to monkey patch the class
    id_map = type(self)._object_id_to_value_map
    return id_map[id(self)]


class _LazyEnumMeta(type):
    def __new__(mcls, name, bases, attrs):
        attrs['_object_id_to_value_map'] = {}
        attrs['_identifier_value_map'] = {}
        attrs['_alias_to_member_map'] = OrderedDict()
        cls = super().__new__(mcls, name, bases, attrs)

        # grab the first parameter name from the __init__ method
        # then inject it to the class as a read-only property
        id_name = cls.__init__.__code__.co_varnames[1]
        cls._identifier_field_name = id_name
        setattr(cls, id_name, property(_get_identifier_value))

        return cls

    def __call__(cls, value, *args, **kwargs):
        # rely on the class to provide the validation method
        if not cls._validate_identifier_value(value):
            raise ValueError(f'Identifier field {cls._identifier_field_name!r} '
                            f'has invalid value {value!r}.')

        # create a new memeber iff no existing member has the same identifier value
        if value not in cls._identifier_value_map:
            new_member = super().__call__(value, *args, **kwargs)
            cls._object_id_to_value_map[id(new_member)] = value
            cls._identifier_value_map[value] = new_member
        elif args or kwargs:
            raise ValueError(f'Member with identifier value {value!r} already exists. '
                            f'Cannont pass additional arguments {args} or {kwargs}.')

        return cls._identifier_value_map[value]

    def __contains__(cls, other):
        return other in cls._identifier_value_map.values()

    def __len__(cls):
        return len(cls._identifier_value_map)

    def __iter__(cls):
        yield from cls._identifier_value_map.values()

    def __setattr__(cls, attr_name, attr_value):
        if attr_name in cls._alias_to_member_map:
            del cls._alias_to_member_map[attr_name]

        # check if we are setting name to a class member
        if attr_value in cls:
            cls._alias_to_member_map[attr_name] = attr_value

        super().__setattr__(attr_name, attr_value)

    def __delattr__(cls, attr_name):
        if attr_name in cls._alias_to_member_map:
            del cls._alias_to_member_map[attr_name]

        super().__delattr__(attr_name)

    @property
    def identifier_value_map(cls):
        return MappingProxyType(cls._identifier_value_map)

    @property
    def alias_to_member_map(cls):
        return MappingProxyType(cls._alias_to_member_map)


class LazyEnum(metaclass=_LazyEnumMeta):
    # the first two methods serve as the defaults if a subclass didn't provide them
    # to avoid error when _LazyEnumMeta attempts to use those two methods
    def _validate_identifier_value(value):
        return True

    def __init__(self, identifier_value):
        pass

    @property
    def identifier_field(self):
        id_name = type(self)._identifier_field_name
        return _Identifier(id_name, getattr(self, id_name))

    @property
    def all_aliases(self):
        pairs = type(self)._alias_to_member_map.items()
        return [alias for alias, member in pairs if member is self]

Questions

1.

The above code doesn't work well with dataclasses. If we write

from lazyenum import LazyEnum
from dataclasses import dataclass

@dataclass
class Product(LazyEnum):
    def _validate_identifier_value(product_id):
        return isinstance(product_id, int) and (1001 <= product_id <= 9999)

    product_id : int
    product_title : str

then type the following in the console:

>>> prod1 = Product(1001, 'First Product')
>>> prod1.product_id = 2001
>>> Product(2001)
TypeError: __init__() missing 1 required positional argument: 'product_title'
>>> Product(1001)
Product(product_id=2001, product_title='First Product')

We can change the product_id but the member is still identified by the old value! How can I fix this?

2. Apart from the issue of dataclasses, is there any problem in the above code? Where can I make improvements?

Answer 1

High-level

1. LazyEnum should be separate from the underlying datatype. You should allow a similar interface like the following:

   @dataclasses.dataclass
   class Product:
       id: int
       name: str
   
   class Products(metaclass=LazyEnumMeta, type=Product):
       pass
   
   Products(1001, 'foo')
   

   This can alleviate the following problem. If, however, you still run into this problem, it's because your using Products for something it shouldn't be used for.

   But be careful, we may accidentally overwrite other class attributes.

2. You should allow the enum class to function similarly to how Python's does.

   class Products(metaclass=LazyEnumMeta, type=Product):
       product_one = first_product = 1001
       product_two = 1002
   

   By only using the ID we can allow the the Product to not be constructed yet, but still define the mapping.

3. You should re-decide where the following exception is handled.

   Member with identifier value {value!r} already exists.

   If you handle it on Products then you could theoretically allow people to create multiple Products with different values that aren't in the enum.

   If you go Graipher's route then you're locking down Product when you may want to use it in two enums.

   It boils down to, where do you want the singleton Products to be scoped?

4. I'm not a fan of having the __call__ function as a __getitem__.

Here's a proof of concept for the above suggestions.

import dataclasses
import inspect


class LazyEnumMeta(type):
    def __new__(mcls, name, bases, attrs, type):
        _attrs = {}
        for key in list(attrs.keys()):
            if not (key.startswith('__') and key.endswith('__')):
                _attrs[key] = attrs.pop(key)

        attrs['_type'] = type
        arguments = inspect.signature(type).parameters.values()
        attrs['_id'] = next(iter(arguments)).name
        attrs['_attrs'] = _attrs
        attrs['_items'] = {}
        return super().__new__(mcls, name, bases, attrs)

    def __call__(self, *args, **kwargs):
        id = args[0] if args else kwargs[self._id]
        if id in self._items:
            return self._items[id]
        self._items[id] = item = self._type(*args, **kwargs)
        return item

    def __getitem__(self, id):
        return self._items[id]

    def __getattr__(self, name):
        return self._items[self._attrs[name]]


@dataclasses.dataclass
class Product:
    id: int
    name: str


class Products(metaclass=LazyEnumMeta, type=Product):
    FIRST = 1001


print(Products(id=1001, name='foo'))
print(Products[1001])
print(Products.FIRST)

Granular

• Many of your names are long and needlessly use Hungarian notation _object_id_to_value_map. This can just be _id_to_value or _by_ids.

  You should notice that many of my names above are really short. type, id, attrs, items. You don't need long names, and when you do it's normally a good idea to refactor the code to prevent a god class.

• I'd much prefer to use inspect than the low level cls.__init__.__code__.co_varnames[1].

• Your hanging indents are not 'correct'. You should change:

  raise ValueError(f'Identifier field {cls._identifier_field_name!r} '
                  f'has invalid value {value!r}.')
  

  To:

  raise ValueError(f'Identifier field {cls._identifier_field_name!r} '
                   f'has invalid value {value!r}.')
  

  Alternately you can change your style and use:

  raise ValueError(
      f'Identifier field {cls._identifier_field_name!r} '
      f'has invalid value {value!r}.'
  )
  

• I don't really see any gain from using id(new_member). If anything it makes the code more annoying.

• I don't think exposing identifier_value_map, alias_to_member_map, identifier_field or all_aliases publicly are good design choices.

Answer 2

This right here is where I would start asking myself if this is a good idea or if there isn't a better way to achieve this:

The first non-self parameter of __init__ is automatically grabbed by the metaclass and cached, so we don't need to set it as an instance attribute. If we write self.product_id = product_id, it would raise an error when we try to initialize a new member.

This behavior is not obvious and it is not even properly documented, because your class does not have a docstring.

Honestly, I don't quite get your enum story, either. At face value you just have a class that you can inherit from to make your class a singleton class depending on the arguments. For this I might use something like this:

class UniqueInstances:
    """Makes sure that each instance exists only once.
    Subclasses must implement __hash__ and __eq__ for this to work.
    Note that new instances are being created,
    but get discarded if they already exist.
    """
    __instances = {}
    def __new__(cls, *args, **kwargs):
        self = super().__new__(cls)
        self.__init__(*args, **kwargs)
        return self.__instances.setdefault(self, self)

    def __hash__(self):
        raise NotImplementedError

    def __eq__(self, other):
        raise NotImplementedError

class Product(UniqueInstances):
    def __init__(self, product_id, name):
        self.product_id = product_id
        self.name = name

    def __hash__(self):
        return self.product_id

    def __eq__(self, other):
        return self.product_id == other.product_id

p1 = Product(1001, "Test")
p2 = Product(1001, "Foo")
print(p1 is p2, p1.name, p2.name)
# True Foo Foo

This is cheating a bit, because it does create a new instance, but it gets discarded if an equal instance already exists. If this still qualifies as lazy, I'm not sure.

It does however have the advantage that it is more obvious what happens, since you have to define __hash__ and __eq__ which is used to determine if an instance already exists.