# The mood of the nation - Twitter trend

This assignment is taken from Berkeley's CS61A page here.

Introduction

In this project, you will develop a geographic visualization of twitter data across the USA. You will need to use dictionaries, lists, and data abstraction techniques to create a modular program.

The map displayed above depicts how the people in different states feel about Texas. python trends.py -m texas. This image is generated by:

• Collecting public Twitter posts (tweets) that have been tagged with geographic locations and filtering for those that contain the "texas" query term,

• Assigning a sentiment (positive or negative) to each tweet, based on all of the words it contains,

• Aggregating tweets by the state with the closest geographic center, and finally

• Coloring each state according to the aggregate sentiment of its tweets. Red means positive sentiment; blue means negative.

Below assignment is phase 3 of this project. Phase 1 can be found here. Phase 2 can be found here.

Phase 3: The Mood of the Nation

States

The name us_states is bound to a dictionary containing the shape of each U.S. state, keyed by its two-letter postal code. You can use the keys of this dictionary to iterate over all the U.S. states.

In this phase, you will write functions to determine the state that a tweet is coming from, group tweets by state, and calculate the average positive or negative feeling in all the tweets associated with a state.

Problem 8 (1 pt). Implement find_closest_state, which returns the two-letter postal code of the state that is closest to the location of a tweet. Use the geo_distance function (provided in geo.py) to calculate the shortest distance in miles between two positions.

When you complete this problem, the doctests for find_closest_state should pass.

python3 trends.py -t find_closest_state


Problem 9 (1 pt). Implement group_tweets_by_state, which takes a list of tweets and returns a dictionary. The keys of the returned dictionary are state names (two-letter postal codes), and the values are lists of tweets that appear closer to that state's center than any other.

When you complete this problem, the doctests for group_tweets_by_state should pass.

python3 trends.py -t group_tweets_by_state


Problem 10 (1 pt). As an exercise, implement most_talkative_state, which returns the state containing the most tweets containing a given term.

When you complete this problem, the doctests for most_talkative_state should pass.

python3 trends.py -t most_talkative_state


Problem 11 (2 pt). Implement average_sentiments. This function takes the dictionary returned by group_tweets_by_state and also returns a dictionary. The keys of the returned dictionary are the state names (two-letter postal codes), and the values are average sentiment values for all the tweets in that state.

If a state has no tweets with sentiment values, leave it out of the returned dictionary entirely. Do not include a state with no sentiment using a zero sentiment value. Zero represents neutral sentiment, not unknown sentiment. States with unknown sentiment will appear gray, while states with neutral sentiment will appear white.

You should now be able to draw maps that are colored by sentiment corresponding to tweets that contain a given term.

python3 trends.py -m sandwich
python3 trends.py -m obama
python3 trends.py -m texas
python3 trends.py -m my life


If you downloaded the small version of the project, you will only be able to map these four terms. If you would like to map any term, you will need to download this Twitter data file and place it in the data directory of your project.

Solution for phase 3:

from data import word_sentiments, load_tweets
from datetime import datetime
from doctest import run_docstring_examples
from geo import us_states, geo_distance, make_position, longitude, latitude
from maps import draw_state, draw_name, draw_dot, wait, message
from string import ascii_letters
from ucb import main, trace, interact, log_current_line

# Phase 3: The Mood of the Nation

def find_closest_state(tweet, state_centers):
"""Return the name of the state closest to the given tweet's location.

Use the geo_distance function (already provided) to calculate distance
in miles between two latitude-longitude positions.

Arguments:
tweet -- a tweet abstract data type
state_centers -- a dictionary from state names to positions.

>>> us_centers = {n: find_center(s) for n, s in us_states.items()}
>>> sf = make_tweet("Welcome to San Francisco", None, 38, -122)
>>> ny = make_tweet("Welcome to New York", None, 41, -74)
>>> find_closest_state(sf, us_centers)
'CA'
>>> find_closest_state(ny, us_centers)
'NJ'
"""
best_distance = None
closest_state = None
for state, centre_position_of_state in state_centers.items():
if best_distance == None:
best_distance =  geo_distance(centre_position_of_state, tweet_location(tweet))
closest_state = state
continue
else:
distance = geo_distance(centre_position_of_state, tweet_location(tweet))
if  distance < best_distance:
best_distance = distance
closest_state = state
return closest_state

def group_tweets_by_state(tweets):
"""Return a dictionary that aggregates tweets by their nearest state center.

The keys of the returned dictionary are state names, and the values are
lists of tweets that appear closer to that state center than any other.

tweets -- a sequence of tweet abstract data types

>>> sf = make_tweet("Welcome to San Francisco", None, 38, -122)
>>> ny = make_tweet("Welcome to New York", None, 41, -74)
>>> ca_tweets = group_tweets_by_state([sf, ny])['CA']
>>> tweet_string(ca_tweets[0])
'"Welcome to San Francisco" @ (38, -122)'
"""
tweets_by_state = {}
USA_states_center_position = {n: find_center(s) for n, s in us_states.items()}
for tweet in tweets:
state_name_key = find_closest_state(tweet, USA_states_center_position)
tweets_by_state.setdefault(state_name_key, []).append(tweet)
return tweets_by_state

def most_talkative_state(term):
"""Return the state that has the largest number of tweets containing term.

>>> most_talkative_state('texas')
'TX'
>>> most_talkative_state('sandwich')
'NJ'
"""
tweets = load_tweets(make_tweet, term)  # A list of tweets containing term
tweets_by_state = group_tweets_by_state(tweets)
talkative_states = us_states.fromkeys(us_states, 0)
for state, tweet_list in tweets_by_state.items():
for tweet in tweet_list:
for word in tweet_words(tweet):
if word == term:
talkative_states[state] += 1

best_count = None
most_talkative_state = None
for state, count_term in talkative_states.items():
if best_count == None:
best_count = count_term
most_talkative_state = state
continue
else:
if count_term > best_count:
best_count = count_term
most_talkative_state = state
return most_talkative_state

def average_sentiments(tweets_by_state):
"""Calculate the average sentiment of the states by averaging over all
the tweets from each state. Return the result as a dictionary from state
names to average sentiment values (numbers).

If a state has no tweets with sentiment values, leave it out of the
dictionary entirely.  Do NOT include states with no tweets, or with tweets
that have no sentiment, as 0.  0 represents neutral sentiment, not unknown
sentiment.

tweets_by_state -- A dictionary from state names to lists of tweets
"""
averaged_state_sentiments = {}
total_sentiment_for_state = 0
count_sentiment = 0
for state, tweet_list in tweets_by_state.items():
for tweet in tweet_list:
sentiment = analyze_tweet_sentiment(tweet)
if not ((sentiment == 0) or (sentiment == None)):
total_sentiment_for_state += sentiment
count_sentiment += 1
if total_sentiment_for_state != 0:
averaged_state_sentiments[state] = (total_sentiment_for_state / count_sentiment)
total_sentiment_for_state = 0
count_sentiment = 0
return averaged_state_sentiments


As per the testing instructions given in the phase 3 problem, this solution is tested accordingly.

1. From a performance perspective, the time complexity of the functions in the above solution are going polynomial (nested for loops). Is it due to selection of list and dict type data models?

2. How can I improve the naming conventions?

Just like in other quite similar question, I find the already existing code slightly weird but I'll comment only on the code you have written.

find_closest_state

• you don't need to call tweet_location(tweet) multiple times. In a better designed code, find_closest_state would probably take a location and not a tweet as a first parameter.

• you don't need that continue as nothing else would happen for that iteration anyway.

• you can call geo_distance in a single place

• you should use is to compare to None as per PEP8.

• once you've taken previous comments into account, you have :

distance = geo_distance(centre_position_of_state, location)
if best_distance is None:
best_distance = distance
closest_state = state
elif distance < best_distance:
best_distance = distance
closest_state = state


which can be written in a more concise way and the code looks like :

def find_closest_state(tweet, state_centers):
best_distance = None
closest_state = None
location = tweet_location(tweet)
for state, centre_position_of_state in state_centers.items():
distance = geo_distance(centre_position_of_state, location)
if best_distance is None or distance < best_distance:
best_distance = distance
closest_state = state
return closest_state


Alternatively, this can (and probably should) be written using the min builtin.

most_talkative_state

Many comments above apply to most_talkative_state which could easily be rewritten using max.

• existing code is weird? – overexchange May 22 '15 at 13:02
• For instance, I find it surprising that it defines find_closest_state a taking a tweet as a parameter instead of a location. – Josay May 22 '15 at 13:07