I'm writing method that takes a pandas iterator and then computes something based on that. The problem is that some of these computations require iterating twice over the data.
For example the normalized variance needs to first iterate over the data to compute the mean, and then iterate again to compute the (normalized) variance with that. I want to have the iterator as input because otherwise I'd need to compute the
select() within the method which is quite some overhead I don't want to deal with.
However, I cannot reset
tableIterator: it does not support
seek(0). Therefore, I'm running with a generator right now. Here's how it looks:
def computeIterativeMean(iterator): # computes the mean given an iterator n, mean = 0, 0 for chunk in iterator: nCurrent = len(chunk) meanCurrent = chunk.mean() mean = (n * mean + nCurrent * meanCurrent)/(n + nCurrent) n += nCurrent return mean def computeIterativeMoment(iteratorGenerator, n=2): # computes some moment given iterator generator mean = computeIterativeMean(iteratorGenerator()) iterator = iteratorGenerator() i, moment = 0, 0 for chunk in iterator: iCurrent = len(chunk) momentCurrent = ((chunk - mean)**n).mean() moment = (i * moment + iCurrent * momentCurrent)/(i + iCurrent) i += iCurrent return moment
And the code is called from outside as
movementsIteratorGenerator = lambda: store.select( 'movements', where=where, columns=myCols, iterator=True, chunksize=chunksize ) mean = computeIterativeMean(movementsIteratorGenerator()) variance = computeIterativeMoment(movementsIteratorGenerator, n=2)