-1
\$\begingroup\$

This assignment is taken from Berkeley's CS61A page here.

enter image description here

Introduction

In this project, you will develop a geographic visualization of twitter data across the USA. You will need to use dictionaries, lists, and data abstraction techniques to create a modular program.

The map displayed above depicts how the people in different states feel about Texas. python trends.py -m texas. This image is generated by:

  • Collecting public Twitter posts (tweets) that have been tagged with geographic locations and filtering for those that contain the "texas" query term,

  • Assigning a sentiment (positive or negative) to each tweet, based on all of the words it contains,

  • Aggregating tweets by the state with the closest geographic center, and finally

  • Coloring each state according to the aggregate sentiment of its tweets. Red means positive sentiment; blue means negative.

Below assignment is phase 3 of this project. Phase 1 can be found here. Phase 2 can be found here.

Phase 3: The Mood of the Nation

States

The name us_states is bound to a dictionary containing the shape of each U.S. state, keyed by its two-letter postal code. You can use the keys of this dictionary to iterate over all the U.S. states.

In this phase, you will write functions to determine the state that a tweet is coming from, group tweets by state, and calculate the average positive or negative feeling in all the tweets associated with a state.

Problem 8 (1 pt). Implement find_closest_state, which returns the two-letter postal code of the state that is closest to the location of a tweet. Use the geo_distance function (provided in geo.py) to calculate the shortest distance in miles between two positions.

When you complete this problem, the doctests for find_closest_state should pass.

python3 trends.py -t find_closest_state

Problem 9 (1 pt). Implement group_tweets_by_state, which takes a list of tweets and returns a dictionary. The keys of the returned dictionary are state names (two-letter postal codes), and the values are lists of tweets that appear closer to that state's center than any other.

When you complete this problem, the doctests for group_tweets_by_state should pass.

python3 trends.py -t group_tweets_by_state

Problem 10 (1 pt). As an exercise, implement most_talkative_state, which returns the state containing the most tweets containing a given term.

When you complete this problem, the doctests for most_talkative_state should pass.

python3 trends.py -t most_talkative_state

Problem 11 (2 pt). Implement average_sentiments. This function takes the dictionary returned by group_tweets_by_state and also returns a dictionary. The keys of the returned dictionary are the state names (two-letter postal codes), and the values are average sentiment values for all the tweets in that state.

If a state has no tweets with sentiment values, leave it out of the returned dictionary entirely. Do not include a state with no sentiment using a zero sentiment value. Zero represents neutral sentiment, not unknown sentiment. States with unknown sentiment will appear gray, while states with neutral sentiment will appear white.

You should now be able to draw maps that are colored by sentiment corresponding to tweets that contain a given term.

python3 trends.py -m sandwich
python3 trends.py -m obama
python3 trends.py -m texas
python3 trends.py -m my life

If you downloaded the small version of the project, you will only be able to map these four terms. If you would like to map any term, you will need to download this Twitter data file and place it in the data directory of your project.

Solution for phase 3:

from data import word_sentiments, load_tweets
from datetime import datetime
from doctest import run_docstring_examples
from geo import us_states, geo_distance, make_position, longitude, latitude
from maps import draw_state, draw_name, draw_dot, wait, message
from string import ascii_letters
from ucb import main, trace, interact, log_current_line

# Phase 3: The Mood of the Nation

def find_closest_state(tweet, state_centers):
    """Return the name of the state closest to the given tweet's location.

    Use the geo_distance function (already provided) to calculate distance
    in miles between two latitude-longitude positions.

    Arguments:
    tweet -- a tweet abstract data type
    state_centers -- a dictionary from state names to positions.

    >>> us_centers = {n: find_center(s) for n, s in us_states.items()}
    >>> sf = make_tweet("Welcome to San Francisco", None, 38, -122)
    >>> ny = make_tweet("Welcome to New York", None, 41, -74)
    >>> find_closest_state(sf, us_centers)
    'CA'
    >>> find_closest_state(ny, us_centers)
    'NJ'
    """
    best_distance = None
    closest_state = None    
    for state, centre_position_of_state in state_centers.items():
        if best_distance == None:
            best_distance =  geo_distance(centre_position_of_state, tweet_location(tweet))
            closest_state = state           
            continue
        else:
            distance = geo_distance(centre_position_of_state, tweet_location(tweet)) 
            if  distance < best_distance:
                best_distance = distance
                closest_state = state           
    return closest_state

def group_tweets_by_state(tweets):
    """Return a dictionary that aggregates tweets by their nearest state center.

    The keys of the returned dictionary are state names, and the values are
    lists of tweets that appear closer to that state center than any other.

    tweets -- a sequence of tweet abstract data types

    >>> sf = make_tweet("Welcome to San Francisco", None, 38, -122)
    >>> ny = make_tweet("Welcome to New York", None, 41, -74)
    >>> ca_tweets = group_tweets_by_state([sf, ny])['CA']
    >>> tweet_string(ca_tweets[0])
    '"Welcome to San Francisco" @ (38, -122)'
    """
    tweets_by_state = {}
    USA_states_center_position = {n: find_center(s) for n, s in us_states.items()}
    for tweet in tweets:
        state_name_key = find_closest_state(tweet, USA_states_center_position)
        tweets_by_state.setdefault(state_name_key, []).append(tweet)
    return tweets_by_state

def most_talkative_state(term):
    """Return the state that has the largest number of tweets containing term.

    >>> most_talkative_state('texas')
    'TX'
    >>> most_talkative_state('sandwich')
    'NJ'
    """
    tweets = load_tweets(make_tweet, term)  # A list of tweets containing term
    tweets_by_state = group_tweets_by_state(tweets)
    talkative_states = us_states.fromkeys(us_states, 0)
    for state, tweet_list in tweets_by_state.items():
        for tweet in tweet_list:
            for word in tweet_words(tweet):
                if word == term:
                    talkative_states[state] += 1

    best_count = None
    most_talkative_state = None
    for state, count_term in talkative_states.items():
        if best_count == None:
            best_count = count_term
            most_talkative_state = state
            continue
        else:
            if count_term > best_count:
                best_count = count_term
                most_talkative_state = state
    return most_talkative_state

def average_sentiments(tweets_by_state):
    """Calculate the average sentiment of the states by averaging over all
    the tweets from each state. Return the result as a dictionary from state
    names to average sentiment values (numbers).

    If a state has no tweets with sentiment values, leave it out of the
    dictionary entirely.  Do NOT include states with no tweets, or with tweets
    that have no sentiment, as 0.  0 represents neutral sentiment, not unknown
    sentiment.

    tweets_by_state -- A dictionary from state names to lists of tweets
    """
    averaged_state_sentiments = {}
    total_sentiment_for_state = 0
    count_sentiment = 0
    for state, tweet_list in tweets_by_state.items():
        for tweet in tweet_list:
            sentiment = analyze_tweet_sentiment(tweet)
            if not ((sentiment == 0) or (sentiment == None)):
                total_sentiment_for_state += sentiment
                count_sentiment += 1
        if total_sentiment_for_state != 0:
            averaged_state_sentiments[state] = (total_sentiment_for_state / count_sentiment)
        total_sentiment_for_state = 0
        count_sentiment = 0
    return averaged_state_sentiments

As per the testing instructions given in the phase 3 problem, this solution is tested accordingly.

  1. From a performance perspective, the time complexity of the functions in the above solution are going polynomial (nested for loops). Is it due to selection of list and dict type data models?

  2. How can I improve the naming conventions?

\$\endgroup\$
1
\$\begingroup\$

Just like in other quite similar question, I find the already existing code slightly weird but I'll comment only on the code you have written.

find_closest_state

  • you don't need to call tweet_location(tweet) multiple times. In a better designed code, find_closest_state would probably take a location and not a tweet as a first parameter.

  • you don't need that continue as nothing else would happen for that iteration anyway.

  • you can call geo_distance in a single place

  • you should use is to compare to None as per PEP8.

  • once you've taken previous comments into account, you have :

    distance = geo_distance(centre_position_of_state, location)
    if best_distance is None:
        best_distance = distance
        closest_state = state
    elif distance < best_distance:
        best_distance = distance
        closest_state = state
    

which can be written in a more concise way and the code looks like :

def find_closest_state(tweet, state_centers):
    best_distance = None
    closest_state = None
    location = tweet_location(tweet)
    for state, centre_position_of_state in state_centers.items():
        distance = geo_distance(centre_position_of_state, location)
        if best_distance is None or distance < best_distance:
            best_distance = distance
            closest_state = state
    return closest_state

Alternatively, this can (and probably should) be written using the min builtin.

most_talkative_state

Many comments above apply to most_talkative_state which could easily be rewritten using max.

\$\endgroup\$
  • \$\begingroup\$ existing code is weird? \$\endgroup\$ – overexchange May 22 '15 at 13:02
  • \$\begingroup\$ For instance, I find it surprising that it defines find_closest_state a taking a tweet as a parameter instead of a location. \$\endgroup\$ – SylvainD May 22 '15 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.