In a data processing and analysis application I have a dataCleaner class that conducts a series of functions that help me to clean raw time series data.
One of the problems I see in my data is that they often include large gaps at the beginning and end. This is due to the occasional corruption of the timestamp, a behaviour that I cannot influence but can lead to arbitrary timings of individual data records.
Imagine an hourly dataset covering November 2011. Sometimes one of the timestamps is corrupted and may end up recording a date of January 2011. Since the data are sorted by date this puts a point at the beginning which needs to be removed. It is possible that this corruption can occur more than once in any given dataset. I need to detect and remove these outliers if they exist.
So I designed this function to trim off contiguous values at each end of the data if the time gap is considered large. My data arrive into this function in the form of two numpy arrays (timestamps and values) and must be filtered together.
@staticmethod def _trimmed_ends(timestamps, values, big_gap = 60*60*24*7): """ Uses timestamps array to identify and trim big gaps at either end of a dataset. The values array is trimmed to match. """ keep = np.ones(len(timestamps), dtype=bool) big_gaps = (np.diff(timestamps) >= big_gap) n = (0, 0) for i in xrange(len(keep)): if big_gaps[i]: keep[i] = False n += 1 else: break for i in xrange(len(keep)): if big_gaps[::-1][i]: keep[::-1][i] = False n += 1 else: break if sum(n) > 0: logging.info("%i points trimmed (%i from beginning, %i from end)" % (sum(n), n, n)) else: logging.info("No points trimmed") return timestamps[keep], values[keep]
Is this a pythonic way to do this? I have been advised that I might want to convert this into an iterator but I'm not sure it that is possible, let alone desirable. As I understand it, I need to attack the array twice, once forwards and once backwards in order to achieve the desired result.