I am working on a script in Python 3.5 that will take in data from several large files. The script takes a while to run, so I am trying to optimize the functions that take the longest. In particular, each file contains around 1 million arrays of around 200 floats.
I need to:
- compute the difference between consecutive values in each array
- find the point at which the difference between values becomes approximately zero (within some limit). Values beyond this point will be discarded.
- find the average value of the discarded data points
So for one array of one data set:
import numpy as np
# the input data is an array of 190 floats
input = array_of_floats
# compute the difference between consecutive floats in the data array
size = len(input)-1
differences = [(input[i+1]-input[i]) for i in np.arange(0,size)]
# reliable standard deviation used to determine discard limit
good_sigma = np.std(differences[20:60])
# get the upper and lower limit
upperbound = 0 + 3*good_sigma
lowerbound = 0 - 3*good_sigma
# find points between upper and lower bound to discard
discard = np.where((lowerbound < differences) & (differences < upperbound))
# find the average value of input data from the first discarded point to
# the end of the input data array
average_discard = np.average(input[np.min(discard):])
# typically, for an array of around 200 floats, the last 60 data points are
# discarded. The rest of the input array is used for other functions in the
# script.
The "differences" step is what takes the longest. For one data set it isn't so bad (~0.01sec), but each file has over 1 million input arrays that I will have to perform this function on. Can someone provide some insight on whether I have written this in the most optimal way? I was also considering using Cython.
Edited to add example input data
array_of_floats =
[ 2.67944336e-02, 4.67507019e+02, 9.42140930e+02,
1.43761255e+03, 1.87599805e+03, 2.36763574e+03,
2.83424146e+03, 3.31971045e+03, 3.76542065e+03,
4.25951953e+03, 4.72111377e+03, 5.20215820e+03,
5.65431445e+03, 6.12919092e+03, 6.60266846e+03,
7.07001367e+03, 7.52144385e+03, 7.98579102e+03,
8.46145410e+03, 8.89987012e+03, 9.37324805e+03,
9.82474414e+03, 1.02881533e+04, 1.07467061e+04,
1.11988945e+04, 1.16621221e+04, 1.21015020e+04,
1.25664697e+04, 1.30272354e+04, 1.34883242e+04,
1.39572393e+04, 1.44017471e+04, 1.48729639e+04,
1.53420791e+04, 1.57918623e+04, 1.62737021e+04,
1.67290801e+04, 1.71610977e+04, 1.76309434e+04,
1.80829941e+04, 1.85269258e+04, 1.89810938e+04,
1.94234805e+04, 1.98656348e+04, 2.03295234e+04,
2.07511074e+04, 2.11939336e+04, 2.16337148e+04,
2.20985977e+04, 2.25441953e+04, 2.29944922e+04,
2.34240684e+04, 2.38671055e+04, 2.42933281e+04,
2.47243613e+04, 2.51660430e+04, 2.55904414e+04,
2.60533496e+04, 2.64807930e+04, 2.69388066e+04,
2.73816992e+04, 2.78312188e+04, 2.82667852e+04,
2.87178203e+04, 2.91476543e+04, 2.95946309e+04,
3.00434082e+04, 3.04553223e+04, 3.08737930e+04,
3.13186289e+04, 3.17753789e+04, 3.21826094e+04,
3.26074180e+04, 3.30163086e+04, 3.34446172e+04,
3.38478047e+04, 3.43073906e+04, 3.47224141e+04,
3.51193594e+04, 3.55621133e+04, 3.59837227e+04,
3.64019414e+04, 3.68227656e+04, 3.72113984e+04,
3.76188047e+04, 3.80122266e+04, 3.84034336e+04,
3.87982461e+04, 3.91839023e+04, 3.96071328e+04,
3.99813711e+04, 4.03818320e+04, 4.07885820e+04,
4.11744141e+04, 4.15733477e+04, 4.19752812e+04,
4.23835117e+04, 4.27578242e+04, 4.31517734e+04,
4.35184570e+04, 4.39032695e+04, 4.42956719e+04,
4.46545352e+04, 4.50035625e+04, 4.53278594e+04,
4.56956250e+04, 4.60475664e+04, 4.63969258e+04,
4.67411914e+04, 4.70916680e+04, 4.73866953e+04,
4.77376367e+04, 4.80478594e+04, 4.83474922e+04,
4.86498945e+04, 4.89277109e+04, 4.92157930e+04,
4.94906094e+04, 4.97758086e+04, 5.00400352e+04,
5.02743164e+04, 5.05048789e+04, 5.07182930e+04,
5.08856797e+04, 5.09154609e+04, 5.09754453e+04,
5.10226016e+04, 5.10574258e+04, 5.10819297e+04,
5.10935078e+04, 5.11123477e+04, 5.10979922e+04,
5.11024531e+04, 5.11305391e+04, 5.11065547e+04,
5.11401133e+04, 5.11246992e+04, 5.11225039e+04,
5.11162930e+04, 5.11391328e+04, 5.11314414e+04,
5.11349570e+04, 5.11212422e+04, 5.11400430e+04,
5.11528672e+04, 5.11323711e+04, 5.11348008e+04,
5.11359648e+04, 5.11285898e+04, 5.11156602e+04,
5.11257812e+04, 5.11293555e+04, 5.11290586e+04,
5.11461758e+04, 5.11537109e+04, 5.11460273e+04,
5.11330156e+04, 5.11367305e+04, 5.11117695e+04,
5.11316523e+04, 5.11255156e+04, 5.11356445e+04,
5.11348477e+04, 5.11349883e+04, 5.11344688e+04,
5.11403672e+04, 5.11524258e+04, 5.11408828e+04,
5.11490430e+04, 5.11274375e+04, 5.11171719e+04,
5.11178750e+04, 5.11357109e+04, 5.11389062e+04,
5.11551289e+04, 5.11504492e+04, 5.11347695e+04,
5.11315117e+04, 5.11602109e+04, 5.11206133e+04,
5.11212422e+04, 5.11430469e+04, 5.11394023e+04,
5.11372461e+04, 5.11174414e+04, 5.11088906e+04,
5.11134766e+04, 5.11321641e+04, 5.11333203e+04,
5.11284531e+04]
array_of_floats
, so we can check to see if any changes we propose actually improve the timing? \$\endgroup\$