What's the fastest way to get "postcodes" for thousands of coordinates (latitudes & longitudes) in Python? [closed]

Question

I have a dataset that contains 750,000 rows. I want to query each row and get the postcodes using the latitudes and longitudes.

Problem:

The code is executing very fast when I query like 100 rows, and it takes like 10min to query 1000 rows. But, when I try to query like 10,000 rows, it takes hours.

Question(s):

Is there any way I can speed up the querying so that the code will execute faster?

OR

Is there any new approach I can take to get a quicker and accurate result?

My code:

#new_df = green_taxi_fare_df.copy()[:20000]

from geopy.geocoders import Nominatim

postcodes = []

geolocator = Nominatim(user_agent = "app_name")

for index, row in new_df.iterrows():
    row_location = geolocator.reverse(f"{row['Pickup_latitude']}, {row['Pickup_longitude']}", timeout = 10)
    
    postcode = row_location.raw['address']['postcode'] if 'postcode' in row_location.raw['address'] else '00000'
    postcodes.append(postcode)
       
new_df['start_postcode'] = postcodes

Answer 1

If you are restricted to finding zipcodes in the US then I would do this a completely different way.

You have 750,000 rows, i.e. longitude and latitude pairs and you want to map those to zip codes, but public data shows there are only 30-odd thousand zipcodes. Public data can map these to lat-long pairs (see https://gist.github.com/erichurst/7882666)

Then with that mapping you can design a function that can find the nearest neighbour to any lat-long pair and assign it the zipcode that corresponds to that point. This can all be done on your local computer with your static data.

This will avoid the slow network access and you you can potentially optimise the function in various different ways. I expect you should be able to get a lookup down to a few ms if not better, meaning completing the task for 750,000 rows should take maybe 20mins as a rough guess. If you can parallelise maybe it takes 1-2 mins?

Here is a very basic, completely un-optimised attempt that finds a zip from 35000 zips in 7ms:

import random

MAP = {}
for i in range(35000):
    MAP.update({i: (random.random(), random.random())})

def manhattan_distance(point, point2):
    return abs(point[0] - point2[0]) + abs(point[1] - point2[1])

def get_zip(lat, long):
    min_distance = 1e9
    nearest_key = -1
    for k, v in MAP.items():
        new_distance = manhatten_distance(v, (lat, long))
        if new_distance < min_distance:
            min_distance = new_distance
            nearest_key = k
    return nearest_key

import time
start = time.time()
key = get_zip(0.5, 0.5)
end = time.time()

print(f"one eval took: {end-start:.4f} seconds")
print(f"zipcode is '{key}'")

# one eval took: 0.0069 seconds
# zipcode is '28395'