A continuation of Memoizing decorator with retries, part 2, and related to https://codereview.stackexchange.com/a/133493/47529. I liked my decorator before, but especially in my original use case of a spotty network connection it makes sens to allow for some delay between attempts. The specifics of this delay should be pretty configurable - I don't want to restrict myself or anyone else to my decision of a good backoff function, so I included the ability to pass a generator that handles everything. Also adds a bunch of docstrings (finally) to make it more understandable. I like the interface right now, and I think with the docstrings it makes more sense, but as always I'd like some feedback.
As a note - I got feedback on the second part after I wrote this question, and in particular w.r.t the dual responsibilities that this has (for both caching and retrying). If you want to comment on that feel free, but I've already considered that and will do so in my next iteration.
import functools
import random
import time
import itertools as it
from collections import namedtuple
def no_backoff():
"""Dummy generator that never delays."""
for delay in it.repeat(0):
yield delay
def doubling_backoff(start=1):
"""Double backoff time, always.
Parameters
==========
start: int, optional
The first delay to use. Defaults to 1.
Yields
======
int
The amount of time to delay.
"""
yield 0
while True:
for delay in it.count(start):
restart = yield 2 * delay
if restart:
break
def exponential_backoff(interval):
"""Exponential backoff algorithm over some interval.
Backs off such that for `n` successive failed attempts the delay
is calculated as `interval * random[0, 2**n-1]`.
Parameters
==========
interval: int
How large of an interval to use.
Yields
======
int
The amount of time to delay.
Notes
=====
https://en.wikipedia.org/wiki/Exponential_backoff
"""
yield 0
while True:
for num_failed in it.count():
delay = interval * random.randint(0, 2**num_failed - 1)
restart = yield delay
if restart:
break
MemoizedData = namedtuple('MemoizedData', 'is_exception value')
class Memoizer:
"""Memoizing class with multiple extra features.
Supports the ability to retry several times by suppressing
certain exceptions, ability to capture and rethrow previously
unsuppressed but detected exceptions, and support for algorithmic
backoff algorithms.
Parameters
==========
retry_times: int, optional
How many times to retry the function before giving up. Defaults
to 0.
suppressed_exceptions: tuple, optional
Which exceptions to suppress and retry on. Defaults to an empty
tuple (no exceptions are suppressed).
capture_exceptions: bool, optional
Whether or not a thrown exception should be remembered and
rethrown if the same arguments are used once again. Does not
apply to suppressed exceptions. Defaults to False.
backoff_gen: generator, optional
Generator that is used to calculate the time to wait between
attempts. Defaults to a generator `no_backoff` which infinitely
yields 0. A generator supplied here is expected to first yield
a meaningless value, and accept a boolean value, i.e.
>>> backoff_gen.send(True)
If `True` is sent then one of two things has happened:
1. The function has been called for the first time - your
algorithm may need to be appropriately initialized.
2. The function has been called successfully - your generator
may need to be reset as appropriate.
If `False` is sent then the algorithm failed, and the backoff
should be adjusted as necessary.
Returns
=======
any
The value returned by the wrapped function
Raises
======
any
The exception raised by the wrapped function (may be cached).
This exception may have been internally suppressed up to
`retry_times - 1` for a given function call.
Notes
=====
The wrapped function has an additional keyword argument added
to it named `__replace` which can be used to ignore any value
or exception that was previously cached.
"""
def __init__(self, retry_times=0, suppressed_exceptions=tuple(),
capture_exceptions=False, backoff_gen=no_backoff()):
self.retry_times = retry_times
self.suppressed_exceptions = suppressed_exceptions
self.capture_exceptions = capture_exceptions
self.backoff_generator = backoff_gen
self._generator_started = False
def _init_backoff_generator(self):
"""Initializes the backoff generator.
If the generator has not been started, gets the first value
from it and discards it. Then informs the generator that the
function has been started.
Notes
=====
Expects that the backoff generator will yield some value that
can be thrown away when initialized, and then handles a boolean
value as described previously.
"""
if not self._generator_started:
next(self.backoff_generator)
self._generator_started = True
self.backoff_generator.send(True)
def _handle_function(self, function, args, kwargs, raise_suppressed=False):
"""Tries to run the function and capture any values.
Parameters
==========
function: callable
The function to be called.
args: list
The function arguments.
kwargs: dict
The function keyword arguments.
raise_suppressed: bool, optional
Whether or not suppressed exceptions should raise. Defaults
to False.
Returns
=======
MemoizedData
Some memoized data of the result of the function
Raises
======
Exception
Any unsuppressed and uncaptured exception
"""
try:
return MemoizedData(False, function(*args, **kwargs))
except self.suppressed_exceptions:
if raise_suppressed:
raise
except Exception as e:
if self.capture_exceptions:
return MemoizedData(True, e)
raise
def __call__(self, function):
"""Actually wrap a function."""
d = {}
@functools.wraps(function)
def wrapper(*args, __replace=False, **kwargs):
self._init_backoff_generator()
key = (args, tuple(sorted(kwargs.items())))
if key not in d or __replace:
for _ in range(self.retry_times - 1):
result = self._handle_function(function, args, kwargs)
if result is not None:
d[key] = result
break
delay = self.backoff_generator.send(False)
time.sleep(delay)
else:
d[key] = self._handle_function(
function, args, kwargs, raise_suppressed=True
)
if d[key].is_exception:
raise d[key].value
else:
return d[key].value
return wrapper
__init__signature would become incomprehensible. \$\endgroup\$