1
\$\begingroup\$

Written in Python 3.6 and I must admit, this might be too far...

Tell me what you think of this code. There must be a better way...

It's (another decorator...) that allows you to compose functions that perform operations on pandas DataFrames.

Kinda like a reduce for performing sequential functions.

from functools import reduce, partial, wraps
import pandas as pd

class Meta(type):
    """This metaclass's purpose is to define behavior for adding it's subtypes
    """

     def __add__(typ, other):
    bases = tuple(sorted([typ, other], key=lambda x: len(x.__mro__), reverse=True))
    return type('{}_{}'.format(typ.__name__, other.__name__), bases, {})


class DefaultFunc(metaclass=Meta):
    """Abstract base class to be the end item of a Func __mro__
    """

    @staticmethod
    def default_func(dataframe):
        return dataframe

    func = default_func

    def __new__(cls, dataframe):
        return dataframe


class Func(DefaultFunc):
    """Wrapper for a DataFrame function. It's doesn't create an object like a normal class def
    this way it behaves more like an actual function
    """
    def __new__(cls, dataframe):
        return reduce(lambda x, y: y.func(x), list(reversed(list(cls.__mro__)[:-1])), dataframe)

    @property
    def __name__(self):
        return self.func.__name__


def depends(func=None, *, on=Func):
    """Wrap a function in a class that resolves dependencies
    for the operation of the wrapped function

    Arguments:
        func -- the function that is being wrapped
        dependencies -- functions that need to have been previously computed
            prior to this wrapped function being executed

    Returns the result of the wrapped function
    """
    if not func:
        return partial(depends, on=on)

    @wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)

    try:
        on = tuple(on)
    except TypeError:
        on = (on,)

    return type(func.__name__, on, {'func': wrapper})

if __name__ == '__main__':

    df = pd.DataFrame([{'a': 1, 'b': 2}, {'c': 3, 'd': 2}])


    @depends
    def example(dataframe):
        dataframe[example.__name__] = dataframe['d'] * 2
        return dataframe


    @depends(on=example)
    def example_2(dataframe):
        dataframe[example.__name__] = dataframe['example'] * 2
        return dataframe

USAGE

df
Out[1]:
     a    b    c    d
0  1.0  2.0  NaN  NaN
1  NaN  NaN  3.0  2.0

example(df)
Out[2]:
     a    b    c    d  example
0  1.0  2.0  NaN  NaN      NaN
1  NaN  NaN  3.0  2.0      4.0

example_2(df)
Out[3]:
     a    b    c    d  example
0  1.0  2.0  NaN  NaN      NaN
1  NaN  NaN  3.0  2.0      8.0

example_3 = example + example_2
example_3(df)
Out[5]:
     a    b    c    d  example
0  1.0  2.0  NaN  NaN      NaN
1  NaN  NaN  3.0  2.0     16.0
\$\endgroup\$
  • \$\begingroup\$ Do you have a slightly more realistic example as to why you want to do this? Maybe I'm just low on coffee, but this doesn't seem obviously useful to me, and having a use-case in mind would help the review. \$\endgroup\$ – Dannnno Aug 28 '17 at 19:57
  • \$\begingroup\$ Yes. When adding glyphs to the visualization library 'bokeh` you specify x= somedata, y = someotherdata. I wanted to be able to say x = the_result_of_this_function, y = someotherdata. And have my program run the function and plug in the result. Sometimes you have two functions you want to pass, and one depends on the other being performed first. That's where this comes in. It hijacks the MRO to keep track of what needs to be done first. \$\endgroup\$ – James Schinner Aug 28 '17 at 22:31
  • \$\begingroup\$ say many functions were passed and they depends on many other functions. You don't want to go through the MRO for every passed function. That's where the __add__ comes is. You can then sum all the passed functions and just have the one function to run. - That's the idea anyway. \$\endgroup\$ – James Schinner Aug 28 '17 at 23:17
  • \$\begingroup\$ Can you post an example where using this is noticeably easier/cleaner than just ordering the statements according to their dependency ordering, or where doing that fails to accomplish the goal? \$\endgroup\$ – Dannnno Aug 29 '17 at 20:50
  • \$\begingroup\$ Otherwise, maybe look into dask? It provides task scheduling for operations on a Pandas-like dataframe. \$\endgroup\$ – Dannnno Aug 29 '17 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.