0
\$\begingroup\$

My goal with this snippet is to create an array of coordinates which in turn is a tuple of 68 elements, area and modified area array for all 10k elements and assign it to the df column.

  • Running the complete code results in 5.3, 4.8, 4.6, 4.5 seconds.
#%%

arrayc = [] #array for array of coordinates
areaAr = [] #area type 1
modifiedarea = [] #area type 2
ts = time.time()

for i in range(10708): #number of files
        f = open(df["filepath"][i], "r") #df has column of filepaths 
        x,y = [], []
        for l in f:
            row = l.split()
            x.append(int(float(row[0]))) #68 pairs are of kind 3.82382323e+02 4.563524234e+02.
            y.append(int(float(row[1]))) #I am taking int rounded off to three digits.
        arrayc.append((x,y))
        f.close()
        #x= arrayc[i][0]
        #y = arrayc[i][1]
        areaAr.append(PolyArea(x[36:41],y[36:41]))
        distance = max(np.abs(x[36]-x[39]),np.abs(x[42]-x[45]))
        modifiedarea.append((PolyArea(x[36:41],y[36:41]))/distance)        

te = time.time()
print(-ts+te)
def PolyArea(x,y):
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

How can I minimise the execution time ?

Updates:

  • File generator code:
import numpy as np 
filepath = []
root = '~/Desktop/test/'
for i in range(10):
    for j in range(68):
        numx = ( np.random.randint(100,200))
        numy = np.random.randint(100,200)
        f = open(root + str(i) + ".txt","a")
        f.write(str(numx) + " " + str(numy) + "\n")
  • The numbers 36 41 etc are the coordinates of the polygon of interest on the image. It is fixed that the polygon will always be marked by these coordinates.
\$\endgroup\$
6
  • \$\begingroup\$ Welcome to CodeReview. Your timings include I/O, which is notoriously difficult to do in an unbiased way. Opening (and, to a lesser extent, reading & closing) a legion of files is going to take its time, sweet or not. Did you repeat the measurement for the first case immediately following one of the others? \$\endgroup\$
    – greybeard
    Aug 3 '19 at 8:45
  • \$\begingroup\$ @greybeard I am currently running the different tests on terminal, instead of integrated jupyter. Can you clarify which order should I run them in? I can do the same in Jupyter if it matters. But the order first. :) \$\endgroup\$
    – aki
    Aug 3 '19 at 9:30
  • \$\begingroup\$ The order should be almost immaterial. Running from an empty block&file cache as opposed to a primed one may make all of a difference: in an automated way, do each of the approaches in turn, about seven "turns". Put the first runs aside and have a look on the variance of the remaining readings. \$\endgroup\$
    – greybeard
    Aug 3 '19 at 10:45
  • 1
    \$\begingroup\$ Please see What (not) to do when someone answers my question? and roll back any changes to the code presented. \$\endgroup\$
    – greybeard
    Aug 3 '19 at 11:23
  • 2
    \$\begingroup\$ "The low-hanging fruits" are not handling a lot of files and not doing things over (like the duplicated call to PolyArea() Georgy spotted, too). \$\endgroup\$
    – greybeard
    Aug 3 '19 at 11:27
2
\$\begingroup\$

I couldn't run your code, so I can't say how much faster the following will be.

My suggestion is to use NumPy's loadtxt function to get the array of the necessary coordinates. With this function, you can specify skiprows and max_rows parameters to get the necessary rows, 36-45. This should be more efficient than reading all the file in memory.

Here:

areaAr.append(PolyArea(x[36:41],y[36:41])) 
distance = max(np.abs(x[36]-x[39]),np.abs(x[42]-x[45]))
modifiedarea.append((PolyArea(x[36:41],y[36:41]))/distance)        

you calculate PolyArea two times, but it is enough to calculate it only once and then reuse the result.

The final code could lool like this:

for filepath in df['filepath'].iloc[:10708]:
    values = np.loadtxt(filepath, 
                        skiprows=35,
                        max_rows=10)
    x = values[:, 0]
    y = values[:, 1]
    area = PolyArea(x[:5], y[:5])
    areaAr.append(area)
    distance = max(np.abs(x[0] - x[3]), np.abs(x[6] - x[9]))
    modifiedarea.append(area / distance)  
\$\endgroup\$
0
0
\$\begingroup\$

For the same PolyArea function, Priority order goes like this:

Order of milliseconds.


def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

df["areaAr"] =   PolyArea(df['filepath'].all())

The above method is fastest that I tried. Below is another with order of single digit seconds.

def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

df["areaAr"] =  df.apply(lambda row: PolyArea(row['filepath']), axis=1)

Below is another moderate method. Still better than crude iteration which goes double digit.

def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))
for index,row in df.iterrows():
        areaAr.append(PolyArea(row["filepath"]))

Source, Medium blog: https://engineering.upside.com/a-beginners-guide-to-optimizing-pandas-code-for-speed-c09ef2c6a4d6

\$\endgroup\$
4
  • \$\begingroup\$ The first code looks strange. pd.Series.all returns True or False, so I'm not sure how you managed to run it without getting an error. The difference in time between the second and the third code, AFAIK, should be minimal as apply also runs a loop under the hood. \$\endgroup\$
    – Georgy
    Aug 3 '19 at 17:00
  • \$\begingroup\$ @Georgy yes, first one is troubling. I didn't verify after I used the .all. I meant to do something like df["areaAr"] = PolyArea(df['landmarkpath']) but got The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). Ik it's offtopic, but I would appreciate directions/edits. I should have taken care of series in PolyArea function. Since all other use loops, only this is different. \$\endgroup\$
    – aki
    Aug 3 '19 at 17:03
  • 1
    \$\begingroup\$ Unfortunately, I don't have experience of improving performance further than what you have right now. I don't think you can squeeze more out of NumPy or pandas here. For some directions, take a look at Dask and Numba, or try out PyPy. I've never tested them myself though. \$\endgroup\$
    – Georgy
    Aug 3 '19 at 17:30
  • \$\begingroup\$ Regrettably, I'm out of my depth, here. \$\endgroup\$
    – greybeard
    Aug 4 '19 at 1:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.