4
\$\begingroup\$

This is a program I wrote to grab restaurants/bars from Google, Yelp, and Foursquare. It then ranks them more effectively based on the rating, the number of ratings, and the number of data sources using a Bayesian average. My guess is that the main method could be broken into more functions. I'm also guessing I'm missing some handy list comprehension tricks. Any suggestions?

main.py

import csv
import time

import foursquare
import yelp
import google


def bayesian(R, v, m, C):
    """
    Computes the Bayesian average for the given parameters

    :param R: Average rating for this business
    :param v: Number of ratings for this business
    :param m: Minimum ratings required
    :param C: Mean rating across the entire list
    :returns: Bayesian average
    """

    # Convert to floating point numbers
    R = float(R)
    v = float(v)
    m = float(m)
    C = float(C)

    return ((v / (v + m)) * R + (m / (v + m)) * C)


def remove_duplicate_names(full_list):
    """
    Fixes issue with multiple API calls returning the same businesses

    :param R: The entire unfiltered list
    :returns: Filtered list
    """

    names = set()
    filtered_list = []
    for business in full_list:
        if business.name not in names:
            filtered_list.append(business)
            names.add(business.name)

    return filtered_list


def main():
    """
    Finds all the bars/restaurants in the given area. Use different
    lat/long points to cover entire town since API calls have length limits.
    """

    input_value = ''
    locations = []

    distance = input('Search Radius (meters): ')
    while input_value is not 'n':
        lat = input('Lat: ')
        lng = input('Long: ')
        locations.append((lat, lng))
        input_value = raw_input('Would you like more points? (y/n) ')

    venues, businesses, places = [], [], []

    for lat,lng in locations:

        # Retrieve all businesses for all sources
        print 'Searching lat: {} long: {} ...'.format(lat, lng)
        venues.extend(foursquare.search(lat, lng, distance))
        businesses.extend(yelp.search(lat, lng, distance))
        places.extend(google.search(lat, lng, distance))

        # Rate-limit API calls
        time.sleep(1.0)

    # Remove duplicates from API call overlap
    venues = remove_duplicate_names(venues)
    businesses = remove_duplicate_names(businesses)
    places = remove_duplicate_names(places)

    # Calculate low threshold and average ratings
    fs_low = min(venue.rating_count for venue in venues)
    fs_avg = sum(venue.rating for venue in venues) / len(venues)

    yp_low = min(business.rating_count for business in businesses)
    yp_avg = sum(business.rating for business in businesses) / len(businesses)

    gp_low = min(place.rating_count for place in places)
    gp_avg = sum(place.rating for place in places) / len(places)

    # Add bayesian estimates to business objects
    for v in venues:
        v.bayesian = bayesian(v.rating, v.rating_count, fs_low, fs_avg)
    for b in businesses:
        b.bayesian = bayesian(b.rating * 2, b.rating_count, yp_low, yp_avg * 2)
    for p in places:
        p.bayesian = bayesian(p.rating * 2, p.rating_count, gp_low, gp_avg * 2)

    # Combine all lists into one
    full_list = []
    full_list.extend(venues)
    full_list.extend(businesses)
    full_list.extend(places)
    print 'Found {} total businesses!'.format(len(full_list))

    # Combine ratings of duplicates
    seen_addresses = set()
    filtered_list = []
    for business in full_list:
        if business.address not in seen_addresses:
            filtered_list.append(business)
            seen_addresses.add(business.address)
        else: 
            # Find duplicate in list
            for b in filtered_list:
                if b.address == business.address:
                    # Average bayesian ratings and update source count
                    new_rating = (b.bayesian + business.bayesian) / 2.0
                    b.bayesian = new_rating
                    b.source_count = b.source_count + 1

    # Sort by Bayesian rating
    filtered_list.sort(key=lambda x: x.bayesian, reverse=True)

    # Write to .csv file
    with open('data.csv', 'w') as csvfile:

        categories = ['Name', 'Rating', 'Number of Ratings', 'Checkins', 'Sources']
        writer = csv.DictWriter(csvfile, fieldnames=categories)

        writer.writeheader()
        for venue in filtered_list:
            writer.writerow({'Name': venue.name.encode('utf-8'),
                             'Rating': '{0:.2f}'.format(venue.bayesian),
                             'Number of Ratings': venue.rating_count,
                             'Checkins': venue.checkin_count,
                             'Sources': venue.source_count})


if __name__ == '__main__':
    main()
\$\endgroup\$
6
\$\begingroup\$

Descriptive names

The function signature is:

bayesian(R, v, m, C)

But then you go a long way describing these single letter parameters in the docstring:

:param R: Average rating for this business
:param v: Number of ratings for this business
:param m: Minimum ratings required
:param C: Mean rating across the entire list

Most usually, descriptive code is preferred over descriptive comments / docstrings for the simple reason that having two things (code / comments) instead of one (code) doubles the maintenance effort, and if code and comments get out of sync, the code becomes extremely confusing.

Built-ins

names = set()
filtered_list = []
for business in full_list:
    if business.name not in names:
        filtered_list.append(business)
        names.add(business.name)

return filtered_list

Becomes:

return list(set(business))

The code does not care about the order of the restaurants as far as I can see, so the fact that set changes order should not be a problem.

Function for input

Getting user input is a detail, when looking at the main structure of the program in main we don't care about it, so just use a function.

while input_value is not 'n':
    lat = input('Lat: ')
    lng = input('Long: ')
    locations.append((lat, lng))
    input_value = raw_input('Would you like more points? (y/n) ')

No input in Python 2

It automatically evaluates the input, it is dangerous to execute anything the user enters and universally considered bad practice. Use int(raw_input(x))

The overload

+ means many things in Python, one of them is adding lists:

full_list = []
full_list.extend(venues)
full_list.extend(businesses)
full_list.extend(places)

Becomes:

full_list = venues + businesses + places

With a clear gain in clarity.

\$\endgroup\$
  • 1
    \$\begingroup\$ Thanks so much. Lots of great suggestions here. I appreciate it! \$\endgroup\$ – leerob Dec 11 '15 at 22:03
  • \$\begingroup\$ On your first point: The main reason I chose comments over descriptive variable names was because of the formula at the end. I think this might be an exception to your otherwise agreed upon rule. That formula is pulled directly from here -> imdb.com/help/show_leaf?votestopfaq \$\endgroup\$ – leerob Dec 11 '15 at 22:10
  • \$\begingroup\$ Yes, mathematical formulas always are a gray area for naming ... I think that including a link to the original source would be the best idea. \$\endgroup\$ – Caridorc Dec 11 '15 at 22:17
1
\$\begingroup\$

I have a few comments in addition to Caridorc's good comments:

  • Within bayesian() you convert to float, but before that you possibly use int – When providing the parameters to this function you do some math, which could or could not be int operations. You might want to enforce the float at an earlier level
  • Change into a list of search engines – Instead of duplicating your logic three times, I would change into storing the results in a list of list, and use a list of providers to keep addresses, search method, name of provider, and so on. This could simplify your logic, and would make it easier to extend to new providers.
  • No input validation – What is the input format for latitude and longitude? I know there exists at least three or four different variants. Which variant is accepted by all of these search engines?
  • Split up into some more functions – I like the way you call main() but I would have split it up into more functions, so it could read something like:

    def main():
       locations = get_location_list()
       restaurants = execute_search(locations, search_engines)
       rated_restaurants = calculate_restaurant_rating(restaurant)
       write_restaurants("data.csv", rated_restaurants)
    
       # Or the ugly version of the same...
       write_restaurants("data.csv", 
         calculate_restaurant-rating(
           execute_search(
             get_location_list(),
             SEARCH_ENGINES
           )
         )
       )
    

    Having this functions defined would allow for your script to be used as a module in its logical parts, and you gather and manipulate the data according to different needs presenting it self. And still you could call it as a script to do a single search.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.