Read coordinates from many files and calculate polygon areas

Question

My goal with this snippet is to create an array of coordinates which in turn is a tuple of 68 elements, area and modified area array for all 10k elements and assign it to the df column.

• Running the complete code results in 5.3, 4.8, 4.6, 4.5 seconds.

#%%

arrayc = [] #array for array of coordinates
areaAr = [] #area type 1
modifiedarea = [] #area type 2
ts = time.time()

for i in range(10708): #number of files
        f = open(df["filepath"][i], "r") #df has column of filepaths 
        x,y = [], []
        for l in f:
            row = l.split()
            x.append(int(float(row[0]))) #68 pairs are of kind 3.82382323e+02 4.563524234e+02.
            y.append(int(float(row[1]))) #I am taking int rounded off to three digits.
        arrayc.append((x,y))
        f.close()
        #x= arrayc[i][0]
        #y = arrayc[i][1]
        areaAr.append(PolyArea(x[36:41],y[36:41]))
        distance = max(np.abs(x[36]-x[39]),np.abs(x[42]-x[45]))
        modifiedarea.append((PolyArea(x[36:41],y[36:41]))/distance)        

te = time.time()
print(-ts+te)
def PolyArea(x,y):
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

How can I minimise the execution time ?

Updates:

• File generator code:

import numpy as np 
filepath = []
root = '~/Desktop/test/'
for i in range(10):
    for j in range(68):
        numx = ( np.random.randint(100,200))
        numy = np.random.randint(100,200)
        f = open(root + str(i) + ".txt","a")
        f.write(str(numx) + " " + str(numy) + "\n")

• The numbers 36 41 etc are the coordinates of the polygon of interest on the image. It is fixed that the polygon will always be marked by these coordinates.

Answer 1

I couldn't run your code, so I can't say how much faster the following will be.

My suggestion is to use NumPy's loadtxt function to get the array of the necessary coordinates. With this function, you can specify skiprows and max_rows parameters to get the necessary rows, 36-45. This should be more efficient than reading all the file in memory.

Here:

areaAr.append(PolyArea(x[36:41],y[36:41])) 
distance = max(np.abs(x[36]-x[39]),np.abs(x[42]-x[45]))
modifiedarea.append((PolyArea(x[36:41],y[36:41]))/distance)        

you calculate PolyArea two times, but it is enough to calculate it only once and then reuse the result.

The final code could lool like this:

for filepath in df['filepath'].iloc[:10708]:
    values = np.loadtxt(filepath, 
                        skiprows=35,
                        max_rows=10)
    x = values[:, 0]
    y = values[:, 1]
    area = PolyArea(x[:5], y[:5])
    areaAr.append(area)
    distance = max(np.abs(x[0] - x[3]), np.abs(x[6] - x[9]))
    modifiedarea.append(area / distance)  

Answer 2

For the same PolyArea function, Priority order goes like this:

Order of milliseconds.

def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

df["areaAr"] =   PolyArea(df['filepath'].all())

The above method is fastest that I tried. Below is another with order of single digit seconds.

def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))

df["areaAr"] =  df.apply(lambda row: PolyArea(row['filepath']), axis=1)

Below is another moderate method. Still better than crude iteration which goes double digit.

def PolyArea(urll):
        values = np.loadtxt(urll, skiprows=35,  max_rows=10)
        x = values[:,0]
        y = values[:,1]
        x = x[:5]
        y = y[:5] 
        return 0.5*np.abs(np.dot(x,np.roll(y,1))-np.dot(y,np.roll(x,1)))
for index,row in df.iterrows():
        areaAr.append(PolyArea(row["filepath"]))

Source, Medium blog: https://engineering.upside.com/a-beginners-guide-to-optimizing-pandas-code-for-speed-c09ef2c6a4d6