3
\$\begingroup\$

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing 156055 rows with unique POINT geometries) as to a geoSeries geometry in dataframe 2 (75 rows POINTS).

Question: How can the following code be optimized so as to make it quicker? As an example, I would love some code that uses the dask distributed dataframes map_partitions solution to apply the below functions on a partitioned part of a (sub)-dataframe so as to speed things up. Optimized vector-based cython code would also be a solution etc..

A peak in both dataframes:

df1:

enter image description here

df2:

enter image description here

In the ah frame (df2) there are 75 points, around which I place a buffer of 1000 meters. In each buffer I calculate the distance (using the points of df1 representing households) and nearest point.

My code so far works but, as said, is slow:

import pandas as pd
import geopandas as gpd
from shapely import wkt, wkb
from shapely.geometry import Polygon, Point, LinearRing
from shapely.ops import nearest_points
from tqdm import tqdm, tqdm_notebook

# crs
crs= {'init': 'epsg:28992'}

def calculate_distance(row, dest_geom, src_col='geometry', 
                  target_col='distance'):
    """
    Calculates distance between single Point geometry and GeoDF 
    with Point geometries.

    Parameters
    ----------
    dest_geom : Point geometry to which distances will be calculated to
    src_col : column with Point objects from where the distances will be calculated from.
    target_col : name of target column where the result will be stored.
    """
    # calculate distances
    dist = row[src_col].distance(dest_geom)
    # to km
    dist_km = dist/1000
    # Assign the distance to the original data
    row[target_col] = dist_km
    return row

def find_nearest_point(row, geom_union, df1, df2, geom1_col='geometry', 
            geom2_col='geometry', src_column=None):
    """Find nearest point,return corresponding value from specified 
    column
    Parameters
    ----------
    geom_union = variable with unary union of points from second frame.
    df1 = dataframe 1 containing geometry column (points)
    df2 = dataframe 2 containing geometry column (points)
    geom1_col = geometry column name from df1
    geom2_col = geometry column name from df2
    src_column =  columns from df2 to be retrieved based on nearest match 
    """
    # Find closest geometry
    nearest = df2[geom2_col] == nearest_points(row[geom1_col], geom_union)[1]
    # Get corresponding value from df2 (based on geometry match)
    value = df2[nearest][src_column].get_values()[0]
    return value

Here i apply the functions with a loop:

n=len(ah)

test = []
for i, row in enumerate(tqdm_notebook(list(ah['buffer'][:n]), 
desc='distance_calc')):
    sub_df = df.loc[(df.geometry.within(ah['buffer'][i])), :]
    sub_df = (sub_df.apply(calculateDistance, 
                       dest_geom= ah['geometry'][i], 
             target_col= 'distance', axis=1))
    print ('shape sub_df {} = {}'.format(i, sub_df.shape))

    indices = (sub_df.apply(nearest, 
                  geom_union=ah.unary_union, 
                  df1=sub_df, 
                  df2=ah, 
                  geom1_col='geometry', 
                  src_column='zaak_id', 
                  axis=1))

    indices_frame = indices.to_frame()
    sub_df = pd.concat([sub_df, indices_frame], axis=1)

    test.append(sub_df)  

test = pd.concat(test, axis=0)
\$\endgroup\$
  • \$\begingroup\$ what if a review of your code points out that the performance could be increased without the use of dask distributed dataframes map_partitions ? would that still be acceptable? \$\endgroup\$ – Malachi Sep 4 '18 at 14:41
  • 1
    \$\begingroup\$ yes. Made that clear now in the question \$\endgroup\$ – ad_s Sep 4 '18 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.