0
\$\begingroup\$

My goal with this snippet is to create an array of coordinates which in turn is a tuple of 68 elements, area and modified area array for all 10k elements and assign it to the df column.

  • Running the complete code results in 5.3, 4.8, 4.6, 4.5 seconds.
#%%

arrayc = [] #array for array of coordinates
areaAr = [] #area type 1
modifiedarea = [] #area type 2
ts = time.time()

for i in range(10708): #number of files
        f = open(df["filepath"][i], "r") #df has column of filepaths 
        x,y = [], []
        for l in f:
            row = l.split()
            x.append(int(float(row[0]))) #68 pairs are of kind 3.82382323e+02 4.563524234e+02.
            y.append(int(float(row[1]))) #I am taking int rounded off to three digits.
        arrayc.append((x,y))
        f.close()
        #x= arrayc[i][0]
        #y = arrayc[i][1]
        areaAr.append(PolyArea(x[36:41],y[36:41]))
        distance = max(np.abs(x[36]-x[39]),np.abs(x[42]-x[45]))
        modifiedarea.append((PolyArea(x[36:41],y[36:41]))/distance)        

te = time.time()
print(-ts+te)
def PolyArea(x,y):
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

How can I minimise the execution time ?

Updates:

  • File generator code:
import numpy as np 
filepath = []
root = '~/Desktop/test/'
for i in range(10):
    for j in range(68):
        numx = ( np.random.randint(100,200))
        numy = np.random.randint(100,200)
        f = open(root + str(i) + ".txt","a")
        f.write(str(numx) + " " + str(numy) + "\n")
  • The numbers 36 41 etc are the coordinates of the polygon of interest on the image. It is fixed that the polygon will always be marked by these coordinates.
\$\endgroup\$
  • \$\begingroup\$ Welcome to CodeReview. Your timings include I/O, which is notoriously difficult to do in an unbiased way. Opening (and, to a lesser extent, reading & closing) a legion of files is going to take its time, sweet or not. Did you repeat the measurement for the first case immediately following one of the others? \$\endgroup\$ – greybeard Aug 3 at 8:45
  • \$\begingroup\$ @greybeard I am currently running the different tests on terminal, instead of integrated jupyter. Can you clarify which order should I run them in? I can do the same in Jupyter if it matters. But the order first. :) \$\endgroup\$ – ankii Aug 3 at 9:30
  • \$\begingroup\$ The order should be almost immaterial. Running from an empty block&file cache as opposed to a primed one may make all of a difference: in an automated way, do each of the approaches in turn, about seven "turns". Put the first runs aside and have a look on the variance of the remaining readings. \$\endgroup\$ – greybeard Aug 3 at 10:45
  • \$\begingroup\$ @greybeard the runtimes are more consistent now. My laptop was under load when I ran the 21 seconds etc tests. A lot of apps were running, jupyter GUI was also less responsive and had a lag. Had to put laptop to sleep, freed up some space and restarted VScode and now the readings are,: around 1 second for area calculations, around 2 -3 for landmarks and total being 4-5, in both Terminal and Jupyter. For minute details, see if anything can be changed in the code itself. \$\endgroup\$ – ankii Aug 3 at 11:11
  • 1
    \$\begingroup\$ "The low-hanging fruits" are not handling a lot of files and not doing things over (like the duplicated call to PolyArea() Georgy spotted, too). \$\endgroup\$ – greybeard Aug 3 at 11:27
2
\$\begingroup\$

I couldn't run your code, so I can't say how much faster the following will be.

My suggestion is to use NumPy's loadtxt function to get the array of the necessary coordinates. With this function, you can specify skiprows and max_rows parameters to get the necessary rows, 36-45. This should be more efficient than reading all the file in memory.

Here:

areaAr.append(PolyArea(x[36:41],y[36:41])) 
distance = max(np.abs(x[36]-x[39]),np.abs(x[42]-x[45]))
modifiedarea.append((PolyArea(x[36:41],y[36:41]))/distance)        

you calculate PolyArea two times, but it is enough to calculate it only once and then reuse the result.

The final code could lool like this:

for filepath in df['filepath'].iloc[:10708]:
    values = np.loadtxt(filepath, 
                        skiprows=35,
                        max_rows=10)
    x = values[:, 0]
    y = values[:, 1]
    area = PolyArea(x[:5], y[:5])
    areaAr.append(area)
    distance = max(np.abs(x[0] - x[3]), np.abs(x[6] - x[9]))
    modifiedarea.append(area / distance)  
\$\endgroup\$
0
\$\begingroup\$

For the same PolyArea function, Priority order goes like this:

Order of milliseconds.


def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

df["areaAr"] =   PolyArea(df['filepath'].all())

The above method is fastest that I tried. Below is another with order of single digit seconds.

def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

df["areaAr"] =  df.apply(lambda row: PolyArea(row['filepath']), axis=1)

Below is another moderate method. Still better than crude iteration which goes double digit.

def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))
for index,row in df.iterrows():
        areaAr.append(PolyArea(row["filepath"]))

Source, Medium blog: https://engineering.upside.com/a-beginners-guide-to-optimizing-pandas-code-for-speed-c09ef2c6a4d6

\$\endgroup\$
  • \$\begingroup\$ The first code looks strange. pd.Series.all returns True or False, so I'm not sure how you managed to run it without getting an error. The difference in time between the second and the third code, AFAIK, should be minimal as apply also runs a loop under the hood. \$\endgroup\$ – Georgy Aug 3 at 17:00
  • \$\begingroup\$ @Georgy yes, first one is troubling. I didn't verify after I used the .all. I meant to do something like df["areaAr"] = PolyArea(df['landmarkpath']) but got The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). Ik it's offtopic, but I would appreciate directions/edits. I should have taken care of series in PolyArea function. Since all other use loops, only this is different. \$\endgroup\$ – ankii Aug 3 at 17:03
  • 1
    \$\begingroup\$ Unfortunately, I don't have experience of improving performance further than what you have right now. I don't think you can squeeze more out of NumPy or pandas here. For some directions, take a look at Dask and Numba, or try out PyPy. I've never tested them myself though. \$\endgroup\$ – Georgy Aug 3 at 17:30
  • \$\begingroup\$ Regrettably, I'm out of my depth, here. \$\endgroup\$ – greybeard Aug 4 at 1:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.