Pythonic data processing? Should I use an iterator?

Question

In a data processing and analysis application I have a dataCleaner class that conducts a series of functions that help me to clean raw time series data.

One of the problems I see in my data is that they often include large gaps at the beginning and end. This is due to the occasional corruption of the timestamp, a behaviour that I cannot influence but can lead to arbitrary timings of individual data records.

Imagine an hourly dataset covering November 2011. Sometimes one of the timestamps is corrupted and may end up recording a date of January 2011. Since the data are sorted by date this puts a point at the beginning which needs to be removed. It is possible that this corruption can occur more than once in any given dataset. I need to detect and remove these outliers if they exist.

So I designed this function to trim off contiguous values at each end of the data if the time gap is considered large. My data arrive into this function in the form of two numpy arrays (timestamps and values) and must be filtered together.

@staticmethod
def _trimmed_ends(timestamps, values, big_gap = 60*60*24*7):
    """
    Uses timestamps array to identify and trim big gaps at either end of a dataset.
    The values array is trimmed to match.
    """
    keep = np.ones(len(timestamps), dtype=bool)
    big_gaps = (np.diff(timestamps) >= big_gap)
    n = (0, 0)
    for i in xrange(len(keep)):
        if big_gaps[i]:
            keep[i] = False
            n[0] += 1
        else:
            break

    for i in xrange(len(keep)):
        if big_gaps[::-1][i]:
            keep[::-1][i] = False
            n[1] += 1
        else:
            break

    if sum(n) > 0:
        logging.info("%i points trimmed (%i from beginning, %i from end)" % (sum(n), n[0], n[1]))
    else:
        logging.info("No points trimmed")
    return timestamps[keep], values[keep]

Is this a pythonic way to do this? I have been advised that I might want to convert this into an iterator but I'm not sure it that is possible, let alone desirable. As I understand it, I need to attack the array twice, once forwards and once backwards in order to achieve the desired result.

Answer 1

Firstly, a loopless solution (not actually tested)

big_gaps = np.diff(timestamps) >= big_gap

As you did

front_gaps = np.logical_and.accumulate(big_gaps)

This produces an array which is True until the first False in big_gaps.

end_gaps = np.logical_and.accumulate(big_gaps[::-1])[::-1]

We do the same thing again, but this time we apply on the reversed array.

big_gaps = np.logical_or(front_gaps, end_gaps)

Now that we have arrays for the front and end portions, combine them

n = np.sum(front_gaps), np.sum(end_gaps)

Take the sums of the gaps arrays, 1 = True, 0 = False to figure out the values for n

keep = np.logical_not(big_gaps)

Invert the logic to figure out which ones to keep

return timestamps[keep], values[keep]

Produce your actual values

A few comments on pieces of your code

@staticmethod
def _trimmed_ends(timestamps, values, big_gap = 60*60*24*7):

I'd make a constant: ONE_WEEK = 60*60*24*7, as I think it would make this clearer

big_gaps = (np.diff(timestamps) >= big_gap)

You don't need those parens

n = (0, 0)

This is a tuple here, but you modify it later. That won't work.

And finally, no you shouldn't use an iterator, for the reasons you mention.