# Using pandas.io.pytables.TableIterator for several iterations

I'm writing method that takes a pandas iterator and then computes something based on that. The problem is that some of these computations require iterating twice over the data.

For example the normalized variance needs to first iterate over the data to compute the mean, and then iterate again to compute the (normalized) variance with that. I want to have the iterator as input because otherwise I'd need to compute the select() within the method which is quite some overhead I don't want to deal with.

However, I cannot reset tableIterator: it does not support seek(0). Therefore, I'm running with a generator right now. Here's how it looks:

def computeIterativeMean(iterator):
# computes the mean given an iterator
n, mean = 0, 0
for chunk in iterator:
nCurrent = len(chunk)
meanCurrent = chunk.mean()
mean = (n * mean + nCurrent * meanCurrent)/(n + nCurrent)
n += nCurrent
return mean

def computeIterativeMoment(iteratorGenerator, n=2):
# computes some moment given iterator generator
mean = computeIterativeMean(iteratorGenerator())

iterator = iteratorGenerator()
i, moment = 0, 0
for chunk in iterator:
iCurrent = len(chunk)
momentCurrent = ((chunk - mean)**n).mean()
moment = (i * moment + iCurrent * momentCurrent)/(i + iCurrent)
i += iCurrent
return moment


And the code is called from outside as

    movementsIteratorGenerator = lambda: store.select(
'movements',
where=where,
columns=myCols,
iterator=True,
chunksize=chunksize
)

mean = computeIterativeMean(movementsIteratorGenerator())
variance = computeIterativeMoment(movementsIteratorGenerator, n=2)