1
\$\begingroup\$

Discovering sk-learn pipelines (https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) made me wonder why weren't I using the same idea for any set of functions applied to a single main argument. Based on this idea I developed the following code (Using kwargs treatment from https://stackoverflow.com/a/58543357):

import inspect
def create_sequential_pipe(func_list):
    """
    Create a pipeline for a series of functions to be applied to a single main argument

    Args:
        func_list: List of functions
    """

    def sequential_pipe(main_arg, **kwargs):

        for func in sequential_pipe._func_list:

            func_args = [k for k, v in inspect.signature(func).parameters.items()]
            func_dict = {k: kwargs.pop(k) for k in dict(kwargs) if k in func_args}
            main_arg = func(main_arg, **func_dict)

        return main_arg

    sequential_pipe._func_list = func_list

    return sequential_pipe


def example_func_1(n, power=1):
    return n ** power


def example_func_2(n, sum=0):
    return n + sum


def example_func_3(n, divide=1):
    return n / divide


if __name__ == "__main__":

    example_pipe= create_sequential_pipe([example_func_1, example_func_2, example_func_3])

    out = example_pipe(2)
    print(out)  # 2.0

    out = example_pipe(2, power=2, sum=1, divide=5)
    print(out)  # 1.0

I feel I could get used to its use, but I couldn't find something similar already implemented in Python, so I feel there must be a downside I am not seeing.

I could just create a wrapper function for the pipeline of functions, but I'm not sure it would be as simple. Is there something similar commonly used in other languages? Do you think it's obfuscates the code? Are there clear downsides to using this?

\$\endgroup\$
4
  • \$\begingroup\$ Hi, I would appreciate if I could know the reason for the downvote, so I might update the question and make it more adequate. Receiving a silent downvote isn't useful. \$\endgroup\$
    – DetBet
    Oct 8, 2020 at 10:36
  • 3
    \$\begingroup\$ Welcome to Code Review, you are correct, whoever down voted should have left a comment. I'm only guessing about why they down voted but there could be 2 reasons, the first is the title and the second is the question seems a little hypothetical due to the function names used. The answer to the question What does the code do? should be clearly stated in either the title or in a text section above the code, then follow that with your concerns about the code. Please see How do I ask a good question for more ideas on improvements. \$\endgroup\$
    – pacmaninbw
    Oct 8, 2020 at 12:09
  • 3
    \$\begingroup\$ That's useful, thanks! I'll try to improve those points \$\endgroup\$
    – DetBet
    Oct 8, 2020 at 13:32
  • \$\begingroup\$ I think your self-answer can safely be un-deleted - any answer that contains at least one material piece of feedback is on-topic. Also, though your question is on the precipice of being closed, I think you've done a good enough job of editing it that it's now basically on-topic. \$\endgroup\$
    – Reinderien
    Oct 9, 2020 at 16:00

1 Answer 1

1
\$\begingroup\$

Without being familiar in use of sklearn, there are still some Python fundamentals that can be improved:

Superfluous comprehension

func_args = [k for k, v in inspect.signature(func).parameters.items()]

can probably be

func_args = inspect.signature(func).parameters.keys()

assuming that parameters has the interface of a regular dictionary. Also, since you're only using it to test membership, wrap it in a call to set().

Set intersection

If you do the above and make func_args a set, then this:

for k in dict(kwargs) if k in func_args

can be replaced by set(kwargs.keys()) & func_args.

Non-repeatable read

There's a potential issue with your use of pop(). This modifies your kwargs in a loop. Are you sure that you want a given kwarg to only apply to one function in your pipe, and not all of them?

Real tests

Convert this:

example_pipe= create_sequential_pipe([example_func_1, example_func_2, example_func_3])

out = example_pipe(2)
print(out)  # 2.0

out = example_pipe(2, power=2, sum=1, divide=5)
print(out)  # 1.0

into an actual unit test - you're already most of the way there, so you might as well.

Variadic function

create_sequential_pipe(func_list):

is a good candidate for being turned into a variadic function, i.e. accepting *func_list. This will make invocation simpler in some cases, including the one you showed at the bottom - there will be no need for an inner list literal.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.