2
\$\begingroup\$

I'm writing method that takes a pandas iterator and then computes something based on that. The problem is that some of these computations require iterating twice over the data.

For example the normalized variance needs to first iterate over the data to compute the mean, and then iterate again to compute the (normalized) variance with that. I want to have the iterator as input because otherwise I'd need to compute the select() within the method which is quite some overhead I don't want to deal with.

However, I cannot reset tableIterator: it does not support seek(0). Therefore, I'm running with a generator right now. Here's how it looks:

def computeIterativeMean(iterator):
    # computes the mean given an iterator
    n, mean = 0, 0
    for chunk in iterator:
        nCurrent = len(chunk)
        meanCurrent = chunk.mean()
        mean = (n * mean + nCurrent * meanCurrent)/(n + nCurrent)
        n += nCurrent
    return mean


def computeIterativeMoment(iteratorGenerator, n=2):
    # computes some moment given iterator generator
    mean = computeIterativeMean(iteratorGenerator())

    iterator = iteratorGenerator()
    i, moment = 0, 0
    for chunk in iterator:
        iCurrent = len(chunk)
        momentCurrent = ((chunk - mean)**n).mean()
        moment = (i * moment + iCurrent * momentCurrent)/(i + iCurrent)
        i += iCurrent
    return moment

And the code is called from outside as

    movementsIteratorGenerator = lambda: store.select(
        'movements',
        where=where,
        columns=myCols,
        iterator=True,
        chunksize=chunksize
    )

    mean = computeIterativeMean(movementsIteratorGenerator())
    variance = computeIterativeMoment(movementsIteratorGenerator, n=2)
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.