1
\$\begingroup\$

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City.
My goal was to output multiple csv files each with 6 fields, Merchant_Name, City, name, formatted_address, latitude, longitude.

For example, if one entry of the csv is Starbucks, Chicago, I want the output csv to contain all the information in the 6 fields (as mentioned above) like so-
Starbucks, Chicago, Starbucks, "200 S Michigan Ave, Chicago, IL 60604, USA", 41.8164613, -87.8127855,
Starbucks, Chicago, Starbucks, "8 N Michigan Ave, Chicago, IL 60602, USA", 41.8164613, -87.8127855
and so on for the rest of the results.

For this, I used Text Search request of Google Maps Places API. Here is what I wrote.

import pandas as pd
# import googlemaps
import requests
# import csv
# import pprint as pp
from time import sleep
import random


def search_output(search):
    if len(data['results']) == 0:
        print('No results found for {}.'.format(search))

    else:

        # Create csv file
        filename = search + '.csv'
        f = open(filename, "w")

        size_of_json = len(data['results'])

        # Get next page token
        # if size_of_json = 20:
            # next_page = data['next_page_token']

        for i in range(size_of_json):
            name = data['results'][i]['name']
            address = data['results'][i]['formatted_address']
            latitude = data['results'][i]['geometry']['location']['lat']
            longitude = data['results'][i]['geometry']['location']['lng']

            f.write(name.replace(',', '') + ',' + address.replace(',', '') + ',' + str(latitude) + ',' + str(longitude) + '\n')

        f.close()

        print('File successfully saved for "{}".'.format(search))

        sleep(random.randint(120, 150))


API_KEY = 'your_key_here'

PLACES_URL = 'https://maps.googleapis.com/maps/api/place/textsearch/json?'


# Make dataframe
df = pd.read_csv('merchant.csv', usecols=[0, 1])

# Construct search query
search_query = df['Merchant_Name'].astype(str) + ' ' + df['City']
search_query = search_query.str.replace(' ', '+')

random.seed()

for search in search_query:
    search_req = 'query={}&key={}'.format(search, API_KEY)
    request = PLACES_URL + search_req

    # Place request and store data in 'data'
    result = requests.get(request)
    data = result.json()

    status = data['status']

    if status == 'OK':
        search_output(search)
    elif status == 'ZERO_RESULTS':
        print('Zero results for "{}". Moving on..'.format(search))
        sleep(random.randint(120, 150))
    elif status == 'OVER_QUERY_LIMIT':
        print('Hit query limit! Try after a while. Could not complete "{}".'.format(search))
        break
    else:
        print(status)
        print('^ Status not okay, try again. Failed to complete "{}".'.format(search))
        break

I want to implement next page token but cannot think of a way which woudn't make it all a mess. Another thing I wish to improve is my csv writing block. And dealing with redundancy.
I further plan to concatenate all the csv files into one(but still keeping the original separate files).

Please note that I'm new to programming, in fact this is actually one of my first programs to achieve something. So please elaborate a bit more when need be. Thanks!

\$\endgroup\$
0
\$\begingroup\$
  • Use the csv library
    • x.writerows(custom_tuple(row) for row in data['results'])
  • Grabbing mutable data from the global scope is, I'm pretty sure, poor form. You should pass data as an argument to search_output.
    • Just pass in data['results'] as results, not the whole data object.
    • Consider calling it output_search_results.
    • Consider using precise type hints.
    • The existing argument search is being used as a file-name, so call it filename.
  • Pass the csv-writer object in as an argument to output_search_results.
    • Maybe this means introducing a middle-layer function to create the csv-writer; that's fine.
    • This is going to let you call output_search_results multiple times after opening the file once.
  • Encapsulate the code that fetches data from google in its own function.
    • Maybe even in its own class, if you're going to try to make more intelligent handlers for the failure cases.
    • Now you can easily build the functionality to fetch a "next page" of results based on the next-page-token. It'll be a sibling to the above function, or maybe an option to it.
  • From here you can either pass data['next_page_token'] as another (optional) parameter to output_search_results, which will call itself recursively, or you can call it in a while loop as you grab each next page.
  • You'll probably need to add some safety measures, such as what to do if a file already exists, etc.
\$\endgroup\$
  • \$\begingroup\$ Thanks for your answer. I'd be pleased if you could fork and send me a PR, here's the GitHub link to my project. \$\endgroup\$ – swingcake Jul 5 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.