4
\$\begingroup\$

I want a generic super-class that I can easily subclass - and if desired add slots to my class - without having to worry about any pickle issues (with the lowest protocol). Is this in general a good idea? And if so, is this class a good implementation?

class HasSlots(object):
    """Top-class that allows mixing of classes with and without slots.

    Takes care that instances can still be pickled with the lowest
    protocol. Moreover, provides a generic `__dir__` method that
    lists all slots.

    """

    # We want to allow weak references to the objects
    __slots__ = ['__weakref__']  

    def _get_all_slots(self):
        """Returns all slots as set"""
        all_slots = (getattr(cls, '__slots__', []) 
                         for cls in self.__class__.__mro__)
        return set(slot for slots in all_slots for slot in slots)

    def __getstate__(self):
        if hasattr(self, '__dict__'):
            # We don't require that all sub-classes also define slots,
            # so they may provide a dictionary
            statedict = self.__dict__.copy()
        else:
            statedict = {}
        # Get all slots of potential parent classes
        for slot in self._get_all_slots():
            try:
                value = getattr(self, slot)
                statedict[slot] = value
            except AttributeError:
                pass
        # Pop slots that cannot or should not be pickled
        statedict.pop('__dict__', None)
        statedict.pop('__weakref__', None)
        return statedict

    def __setstate__(self, state):
        for key, value in state.items():
            setattr(self, key, value)

    def __dir__(self):
        result = dir(self.__class__)
        result.extend(self._get_all_slots())
        if hasattr(self, '__dict__'):
            result.extend(self.__dict__.keys())
        return result

Thus, I can simply sub-class this without worrying about slots and pickle:

class MyClass(HasSlots):
    def __init__(self, x,y):
        self._x = x
        self._y = y

but later maybe I want to optimize it and simply add some slots:

class MySlotClass(HasSlots):
    __slots__ = ['_x', '_y']
    def __init__(self, x,y):
        self._x = x
        self._y = y

And I still want to guarantee that my new object can be pickled:

import pickle
pickle.loads(pickle.dumps(MyClass(1,2))
pickle.loads(pickle.dumps(MySlotClass(1,2))

Are there any serious issues with my idea and implementation? Is this: all_slots = (getattr(cls, '__slots__', []) for cls in self.__class__.__mro__) a good approach to take care that all slots are properly pickled?

Is the approach safe for sub-classes that reference each other in a circular fashion?

\$\endgroup\$
  • \$\begingroup\$ Can you specify Python 2 vs Python 3 (or both)? Also, if you're on Python 2 have you considered using a higher pickle protocol version? \$\endgroup\$ – Veedrac Mar 28 '15 at 10:38
  • \$\begingroup\$ Basically I'm talking about this: stackoverflow.com/questions/3522765/python-pickling-slots-error. Is that the issue you're trying to get around? \$\endgroup\$ – Veedrac Mar 28 '15 at 10:45
  • 1
    \$\begingroup\$ The class should work with Python 2.6, 2.7, 3.3 and 3.4. Moreover, I have to use the lowest Pickle Protocol because the instances will be used in a multiprocessing environment and fork is not always available. \$\endgroup\$ – SmCaterpillar Mar 28 '15 at 10:46
  • \$\begingroup\$ Can you explain what the problem with higher pickle versions and fork is? It's not too clear to me. Are you talking about multiprocessing? It seems to use HIGHEST_PROTOCOL: github.com/python/cpython/blob/2.6/Modules/_multiprocessing/…. \$\endgroup\$ – Veedrac Mar 28 '15 at 10:58
  • \$\begingroup\$ Yeah, you're right I fell for this one: stackoverflow.com/questions/7920601/…. Anyhow, this is still useful if I don't want to write getstate and setstate in every subclass, isn't it? \$\endgroup\$ – SmCaterpillar Mar 28 '15 at 14:09
1
\$\begingroup\$

A few of trivial points:

  • getattr(cls, '__slots__', []) would be better as getattr(..., ()) - tuples are immutable so are normally cached by the implementation.

  • set(slot for slots in all_slots for slot in slots) would be better as {slot for slots in all_slots for slot in slots}

  • It might be better to do

    try:
        statedict = vars(self).copy()
    except TypeError:
        statedict = {}
    
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.