5
\$\begingroup\$

The goal is to have a function that can transform any python function into a curried version of this function. The code is available here, I'll only update the code on repl.it so everyone can make sense of all the remarks and comments made on the code I'll post on codereview.

To be precise, consider this code:

def f(x, y, z, info=None):
  if info:
    print(info, end=": ")
  return x + y + z

g = curry(f)
print("g(2, 3, 4) = ", g(2, 3, 4))
print("g(2, 3)(4) = ", g(2, 3)(4))
print("g(2)(3)(4) = ", g(2)(3)(4))
print("g(2)(3, 4) = "g(2)(3, 4))
print(g(2, info="test A")(3, 4))
print(g(2, info="test A")(3, 4, info="test B"))
print(g(2,3,4)(5))

The goal is to have the following output:

g(2, 3, 4) = 9
g(2, 3)(4) = 9
g(2)(3)(4) = 9
g(2)(3, 4) = 9
test A: 9
test B: 9

And throw an exception on the last line.

My current implementation of curry is the following, I simply wonder if this is as simple as it can be.

from copy import copy
from inspect import signature
import functools


class curry:
    def __init__(self, function):
      self._partial = functools.partial(function)

    def __call__(self, *args, **kwargs):
      partial = functools.partial(self._partial, *args, **kwargs)
      sign = signature(partial.func)
      try:
        sign.bind(*partial.args, **partial.keywords)
        output = partial()
        return output
      except TypeError as e:
        return curry(copy(partial))

    def __str__(self):
      text = "partial: "
      text += str(self._partial.func.__name__ )
      text += str(self._partial.args)
      text += str(signature(self._partial))
      return text

I added an __str__ method that is useful when you want to debug with partials. For example print(g(2,3)) prints:

partial: f(2, 3)(z, info=None)

Any suggestion to improve this is welcome.

Edit:

If overriding partial's string representation isn't needed then a simple function decorator could be used:

from copy import copy
from inspect import signature
import functools


def curry(function):

  def inner(*args, **kwargs):
    partial = functools.partial(function, *args, **kwargs)
    sign = signature(partial.func)
    try:
      sign.bind(*partial.args, **partial.keywords)
      return partial()
    except TypeError as e:
      return curry(copy(partial))

  return inner
\$\endgroup\$
  • 1
    \$\begingroup\$ What's wrong with partial() that you need to use sign.bind to generate a TypeError instead? (Or am I mistakenly considering the purpose of that signature?) \$\endgroup\$ – Mathias Ettinger Feb 24 at 7:37
  • 1
    \$\begingroup\$ The problem is explained in details here. Basically you want to make the difference between partial raising an internal error and partial raising an error because we passed the wrong arguments. \$\endgroup\$ – cglacet Feb 24 at 22:34
  • \$\begingroup\$ Absolutely. This is the kind of things I had in mind, but since you're not making a difference (as everything is in the same try) I wanted to be sure. \$\endgroup\$ – Mathias Ettinger Feb 24 at 22:39
  • \$\begingroup\$ That was totally my mistake, I've thought about that at first then I totally forgot about it. \$\endgroup\$ – cglacet Feb 24 at 22:43
5
\$\begingroup\$

This is interesting! Very good first stab at this.

I'm going to recommend rewriting a lot of this, but I want to call out an important mistake in your code that hides a serious issue. It's your try/except block:

try:
    sign.bind(*partial.args, **partial.keywords)
    return partial()
except TypeError as e:
    return curry(copy(partial))

Because your return partial() is inside the try, if actually calling the function produces a TypeError (consider abs('a')), you suppress it and continue to curry. This is incorrect; you only want to catch TypeError from sign.bind(...)

try:
    sign.bind(*partial.args, **partial.keywords)
except TypeError as e:
    return curry(copy(partial))
else:
    return partial()

Okay, now onto the deeper issues:

Currying in Python is tricky (if not maybe undefined in some cases) because of optional args and kwargs. And to complicate things your "syntax" for it is inconsistent.

Consider your f. While you can do something like:

curry(f)(2, 3, info='A')(4)
curry(f)(2, 3)(info='A')(4)
curry(f)(2)(3)(info='A')(4)

You can't do:

curry(f)(2, 3, 4)(info='A')
curry(f)(2)(3, 4)(info='A')
curry(f)(2)(3)(4)(info='A')

Because providing the 3rd argument (4) you are satisfying the function signature despite perhaps wanting to provide info='A'.

There isn't a good way to work around this. Your curry also behaves unexpectedly with *args or **kwargs functions. Namely, they don't really curry. You can only call them once:

def foo(*args):
    print(args)

def bar(**kwargs):
    print(kwargs)

# While we can do this...
curry(foo)(1, 2, 3)
curry(bar)(a=1, b=2, c=3)

# We can't do this, which is perhaps confusing
curry(foo)(1)(2)(3)
curry(bar)(a=1)(b=2)(c=3)

Consider also curry(list) or curry(set).

Knowing that this won't work relies on you understanding the inner workings of the function you are calling. Some functions opt to take advantage of Python's dynamism and use args/kwargs to accept several different signatures. Some functions are also much more complicated than they let on or are builtin, which leads to the next problem:

inspect.signature doesn't work for some builtins (which are very valid candidates for currying):

>>> signature(map)
ValueError: no signature found for builtin type <class 'map'>
>>> signature(filter)
ValueError: no signature found for builtin type <class 'filter'>

While some support it, support is flaky and something unintuitive.

>>> builtin_functions = [x for x in dir(__builtins__) if x[0] == x[0].lower() and x[0] != '_']
>>> builtin_functions
['abs', 'all', 'any', 'ascii', 'bin', 'bool', 'breakpoint', 'bytearray', 'bytes', 'callable',
 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir',
 'divmod', 'enumerate', 'eval', 'exec', 'exit', 'filter', 'float', 'format', 'frozenset',
 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance',
 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min',
 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr',
 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super',
 'tuple', 'type', 'vars', 'zip']

>>> def supports_signature(f):
...   try:
...     signature(f)
...     return True
...   except ValueError:
...     return False
...
>>> [x for x in builtin_functions if not supports_signature(getattr(__builtins__, x))]
['bool', 'breakpoint', 'bytearray', 'bytes', 'classmethod', 'dict', 'dir', 'filter', 'frozenset',
 'getattr', 'int', 'iter', 'map', 'max', 'min', 'next', 'print', 'range', 'set', 'slice',
 'staticmethod', 'str', 'super', 'type', 'vars', 'zip']

A quick look over this also makes me suspect that some of these signatures may not be correct (or rather, they don't encompass all possible signatures of a function).

Here's another problem: your implementation continues to curry if you pass too many args.

>>> def foo(a, b):
...   print('foo', a, b)
...
>>> curry(foo)(1, 2, 3)
<function curry.<locals>.inner at 0x10ec2ee18>

This is almost certainly going to bite you. You definitely want a type error here like you would get if you called it normally:

>>> foo(1, 2, 3)
TypeError: foo() takes 2 positional arguments but 3 were given

Of course, handling this conflicts with allowing *args (or at least makes it more complicated).

All of these problems compounding, I'd say it may not be wise to use such a pattern for anything more than experimentation or a functional exercise.

But let's say you still want to. Hopefully, the above illustrates that you need to greatly simplify curry for it to be at all useful. Because after all, if something is confusing to implement, unless you have been exceedingly clever, the API you expose probably will also be confusing.

With that out of the way, given all of these issues, I'd recommend placing the following reasonable restriction on your currying (which closely matches currying in functional languages):

You cannot curry a function with default arguments, *args, or **kwargs

Implicitly, this includes things that you can't call signature on. I'd recommend one exception to this: required keyword-only arguments.

def foo(a, *, b, c):
    pass

This doesn't have the same issue as optional kwargs, because you must provide b and c (but you must do so as b=2, c=3, instead of positionally):

foo(1, b=2, c=3)

But note that you cannot include positional arguments after these required kwargs, so we should additionally add this restriction to our currying:

>>> foo(b=2, c=3, 1)
SyntaxError: positional argument follows keyword argument

Also note that we still want to be able to pass any arg as a kwargs (as long as we don't then later pass a positional arg).

Such an approach also allows us to easily check if we need to raise TypeError if too many args are provided.

In keeping with this idea of simplicity, we can also avoid the (unnecessary) call to sign.bind() and signature on every call to the curried function and eliminate partial entirely.

There's one consequence of this approach, though. It requires to reimplement a lot of Python's calling logic. Why? To know if we have been passed too many parameters we have to keep track of what parameters have been passed. And to do that we need to understand what valid combinations of parameters can be passed. This is the only sane approach, though, in my opinion because otherwise you'll get strange errors (likely far away from their source) if curry did not immediately return an error when an invalid parameter was provided (a keyword arg that doesn't exist or too many positionals, for example).

That said, I got a bit nerdsniped by this so I went ahead and implemented all of that logic :) My approach is decently commented and has doctests, I've posted a gist of it so if I plan to update tests, etc. I'll update it there:

from collections import OrderedDict
from inspect import signature, Parameter
from itertools import chain


def curry(func, *args, **kwargs):
    """
    Allows a callable to be passed arguments incrementally, executing it only
    once all arguments have been provided--in the style of functional currying.

    To avoid corner cases, callables with *args, **kwargs, or default arguments
    cannot be curried. This includes some builtins that have multiple
    signatures.

    >>> def print_three(a, b, c):
    ...     print(a, b, c)

    >>> curry(print_three, 1, 2, 3)
    1 2 3
    >>> curry(print_three)(1, 2, 3)
    1 2 3
    >>> curry(print_three)(1)(2, 3)
    1 2 3
    >>> curry(print_three)(1)(2)(3)
    1 2 3
    >>> curry(print_three)(1, 2)(3)
    1 2 3
    >>> curry(print_three)()(1)()(2)()(3)
    1 2 3
    >>> curry(print_three)(1, 2)(3, 4)
    Traceback (most recent call last):
      ...
    TypeError: print_three() takes 3 positional arguments but 4 were given
    >>> curry(print_three)(1, 2)(c=3)
    1 2 3
    >>> curry(print_three)(1)(c=3)(b=2)
    1 2 3
    >>> curry(print_three)(1, 2)(d=3)
    Traceback (most recent call last):
      ...
    TypeError: print_three() got an unexpected keyword argument 'd'

    >>> curry(curry(print_three)(1))(2, 3)
    1 2 3

    >>> def required_keyword(a, b, *, c):
    ...     print(a, b, c)

    >>> curry(required_keyword)(1, 2, 3)
    Traceback (most recent call last):
      ...
    TypeError: required_keyword() takes 2 positional arguments but 3 were given
    >>> curry(required_keyword)(1, 2, c=3)
    1 2 3
    >>> curry(required_keyword)(1, 2)(c=3)
    1 2 3
    >>> curry(required_keyword)(c=3)(a=1, b=2)
    1 2 3

    >>> def has_starargs(a, *args):
    ...     pass
    >>> curry(has_starargs)
    Traceback (most recent call last):
      ...
    TypeError: cannot curry a function with *args or **kwargs

    >>> def has_kwargs(a, **kwargs):
    ...     pass
    >>> curry(has_kwargs)
    Traceback (most recent call last):
      ...
    TypeError: cannot curry a function with *args or **kwargs

    >>> def has_default(a, b=1):
    ...     pass
    >>> curry(has_default)
    Traceback (most recent call last):
      ...
    TypeError: cannot curry a function with default arguments
    """
    # Cannot curry an already curried function since our __call__ has *args
    # and **kwargs, which violates our currying rules.
    if isinstance(func, _curry):
        # Since curry objects are immutable, we can return the same curry
        return func

    params = signature(func).parameters

    if any(_is_star_param(param) for param in params.values()):
        raise TypeError('cannot curry a function with *args or **kwargs')

    if any(param.default != Parameter.empty for param in params.values()):
        raise TypeError('cannot curry a function with default arguments')

    curried = _curry(func, params, (), OrderedDict())

    if args or kwargs:
        return curried(*args, **kwargs)

    return curried


class _curry:
    def __init__(self, func, remaining_params, args, kwargs):
        self._func = func
        self._remaining_params = remaining_params
        self._args = args
        self._kwargs = kwargs

    def __call__(self, *args, **kwargs):
        if not args and not kwargs:
            return self

        if self._kwargs and args:
            raise SyntaxError('positional argument follows keyword argument')

        # Ensure we haven't been passed too many positional arguments
        remaining_params_iter = iter(self._remaining_params.items())

        try:
            for _, (_, expected) in _zip_first(args, remaining_params_iter):
                if not _is_positional_param(expected):
                    raise self._positional_error(len(args))
        except ShortIteratorError:
            raise self._positional_error(len(args))

        # _zip_first will have consumed all of the positional arguments passed.
        # What remains is the positional and keyword argument that haven't been
        # provided.
        new_remaining_params = OrderedDict(remaining_params_iter)

        # Ensure all passed keyword arguments are expected (and eliminate all
        # remaining parameters that are passed)
        for name in kwargs:
            try:
                del new_remaining_params[name]
            except KeyError:
                raise self._type_error(f'got an unexpected keyword argument '
                                       f'\'{name}\'')

        # If all arguments have been provided, call then function
        new_args = self._args + args
        new_kwargs = OrderedDict(chain(self._kwargs.items(), kwargs.items()))

        if not new_remaining_params:
            return self._func(*new_args, **new_kwargs)

        # Otherwise, add the new arguments and return a new curryable function
        return self.__class__(self._func, new_remaining_params, new_args,
                              new_kwargs)

    def _positional_error(self, extra_given):
        remaining_positional = filter(_is_positional_param,
                                      self._remaining_params.values())
        expected = len(self._args) + len(list(remaining_positional))
        s = 's' if expected != 1 else ''

        given = len(self._args) + extra_given

        return self._type_error(f'takes {expected} positional argument{s} but '
                                f'{given} were given')

    def _type_error(self, msg):
        return TypeError(f'{self._func.__name__}() {msg}')


def _is_star_param(param):
    return param.kind in (Parameter.VAR_POSITIONAL, Parameter.VAR_KEYWORD)


def _is_positional_param(param):
    return param.kind in (Parameter.POSITIONAL_ONLY,
                          Parameter.POSITIONAL_OR_KEYWORD)


def _zip_first(first, *rest):
    """Zips arguments until the first iterator is consumed.

    Raises ShortIteratorError if any of the other iterators stop before the
    first is finished.

    >>> list(_zip_first([1, 2], [3, 4, 5]))
    [(1, 3), (2, 4)]

    >>> list(_zip_first([1, 2], [3]))
    Traceback (most recent call last):
      ...
    curry.ShortIteratorError: iterator unexpectedly stopped
    """
    first = iter(first)
    rest = tuple(map(iter, rest))

    for item in first:
        other_items = tuple(map(next, rest))
        if len(other_items) != len(rest):
            raise ShortIteratorError()

        yield (item, *other_items)


class ShortIteratorError(Exception):
    """
    Signals that one of the other iterators ended before the first in a call to
    _zip_first.
    """
    def __init__(self):
        super().__init__('iterator unexpectedly stopped')
\$\endgroup\$
  • \$\begingroup\$ I wanted to comment here with some code, but there is no way to comment properly in here, so answered here. Stackechange is not really optimal for code review to be honest. \$\endgroup\$ – cglacet Feb 24 at 22:48
1
\$\begingroup\$

Nice code - I might reuse that pattern!

Some suggestions for the class implementation:

  • I would recommend formatted string literals in Python 3.6+ over concatenation.
  • A linter such as flake8 will suggest some changes to make your code more pythonic, such as making the class name CamelCase.
  • What causes a TypeError and why do you throw it away?
  • output might as well be inlined.
  • Rather than the ambiguously named sign variable I would prefer signature = inspect.signature(…).
\$\endgroup\$
  • \$\begingroup\$ I wanted to avoid CamelCasing as this was supposed to look like a function for the user. TypeError is raised when the argument supplied to the partial are not sufficient to actually apply the function. "output might as well be inlined" sure thing, if you notice I did it in the function version. \$\endgroup\$ – cglacet Feb 24 at 1:20
  • \$\begingroup\$ I'll update the code on repl.it as remarks come and let everything here as is, so this thread still makes sense to new people coming here. \$\endgroup\$ – cglacet Feb 24 at 1:23
0
\$\begingroup\$

This is a long comment to this post.

Because providing the 3rd argument (4) you are satisfying the function signature despite perhaps wanting to provide info='A'.

Yes I desperately tried to find a way to make a class instance return a value when "called" directly, that would allow to encapsulate both the result and the partial in a class, which would then permit to have g(2, 3, 4) to output the result of the call f(2, 3, 4) and g(2, 3, 4)(info="x") to output the result of the call f(2, 3, 4, info="x"). But I couldn't find a way, I actually had no idea of what to search for (I still don't).

There isn't a good way to work around this. Your curry also make behave unexpectedly with *args or **kwargs functions.

To be honest I didn't even though about this, I don't even know what curry(bar)(1)(2)(3) should output when applied on:

def bar(**kwargs):
    print(kwargs)

inspect.signature doesn't work for some builtins

Hmm, ok, that's very interesting, doesn't this by itself make inspect.signature broken? Shouldn't this be considered to be a bug in python? Is there another (saffer) way of knowing the number of arguments a function takes?

your implementation continues to curry if you pass too many args.

I missed that, I think this should fix it (but I'm using the signature again):

partial = functools.partial(self._partial, *args, **kwargs)
signature = inspect.signature(partial.func)
try:
  signature.bind(*partial.args, **partial.keywords)
except TypeError as e:
  if len(partial.args) < len(signature.parameters):
    return curry(copy(partial))
return partial.func(*partial.args)

Which also has the advantage of having the correct exception raised:

File "main.py", line 28, in test_currying
    print(g(2)(3, 4, 5, 6))
File "main.py", line 43, in __call__
    return partial.func(*partial.args)
TypeError: f() takes from 3 to 4 positional arguments but 5 were given

All of these problems compounding, I'd say it may not be wise to use such a pattern for anything more than experimentation or a functional exercise.

I should have said that in the first place, but yes the only point of this is to better understand how python work.

def foo(a, *, b, c):
   pass

This doesn't have the same issue as optional kwargs, because you must provide b and c (but you must do so as b=2, c=3, instead of positionally)

I wasn't aware of that form, that's very handy. If I understand correctly, this is more or less equivalent to:

def foo(a, b=None, c=None):
  if b is None or c is None:
    raise TypeError("foo is missing some arguments")
  pass

I'll update this with more questions once I'll have read your code. Anyway, thanks for this amazing feedback.

\$\endgroup\$
  • \$\begingroup\$ Certainly :) I'll address in order. (1) Yeah unfortunately there isn't a way to have something that proxies the function call result (but only produces it when required) and also allows more calls. You just can't do that in python. (2) The example you gave would fail because (2)(3) is not **kwargs it's *args. But neither of them work well with currying. (3) No signature isn't broken, some functions are just a little magic (multipurpose) and have more than one signature depending on how you use them (although some of the broken ones like filter don't make sense). \$\endgroup\$ – Bailey Parker Feb 24 at 22:59
  • \$\begingroup\$ (4) Your solution to solve accepting too many params breaks (non-optional) kwarg support. Also what if you had def foo(a, b): and then did curry(foo)(1, 2)(a=1, b=2). That should fail when you try a=1. This is why you must check manually to detect the error as soon as it occurs. Once an argument error has happened you can never reach a state where the function can be run (has enough args) and you get the proper error. (5) Seems reasonable (6) Yes basically. Although foo(a, b=None, c=None) still allows you to do foo(1, 2, 3). You also don't need the pass \$\endgroup\$ – Bailey Parker Feb 24 at 23:05
  • \$\begingroup\$ Hmm, the more you explain your solution, the more I'm convinced that you are right and you have to do almost everything by hand to offer a clean API. I wouldn't have guessed this would be such a hard/complete exercise. I'll keep on reading your code, if you don't see me coming back tomorrow that means you killed me with your magic xD. I feel like I'm reading a python interpreter code. \$\endgroup\$ – cglacet Feb 24 at 23:15
  • \$\begingroup\$ Your code is actually so well written I don't even have much questions. Maybe the only thing I really don't understand is why do you use self.__class__ instead of just _curry on line 149? That being said I still need a bit more time to digest lines 120-125 to make sure I really understand what's going on. \$\endgroup\$ – cglacet Feb 24 at 23:29
  • \$\begingroup\$ :) Ah, that's so if you ever wanted to rename _curry you wouldn't have to go in and replace all the times it references itself. \$\endgroup\$ – Bailey Parker Feb 24 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.