# Assigning sentiment to each tweet - Twitter trend

Below assignment is taken from here.

Introduction

In this project, you will develop a geographic visualization of twitter data across the USA. You will need to use dictionaries, lists, and data abstraction techniques to create a modular program. Below is phase 1 of this project.

Phase 1: The Feelings in Tweets

In this phase, you will create an abstract data type for tweets, split the text of a tweet into words, and calculate the amount of positive or negative feeling in a tweet.

Tweets

First, you will implement an abstract data type for Tweets. The constructor make_tweet is defined at the top of trends.py. make_tweet returns a python dictionary with the following entries:

  text:      a string, the text of the tweet, all in lowercase
time:      a datetime object, when the tweet was posted
latitude:  a floating-point number, the latitude of the tweet's location
longitude: a floating-point number, the longitude of the tweet's location


Problem 1 (1 pt). Implement the tweet_words and tweet_time selectors. Call extract_words to list the words in the text of a tweet.

Problem 2 (1 pt). Implement the tweet_location selector, which returns a position. Positions are another abstract data type, defined at the top of geo.py. Make sure that you understand how to manipulate positions; they play an important role in this project.

When you complete problems 1 and 2, the doctest for make_tweet should pass.

python3 trends.py -t make_tweet


Problem 3 (1 pt). Improve the extract_words function as follows: Assume that a word is any consecutive substring of text that consists only of ASCII letters. The string ascii_letters in the string module contains all letters in the ASCII character set. The extract_words function should list all such words in order and nothing else.

When you complete this problem, the doctest for extract_words should pass.

python3 trends.py -t extract_words


Problem 4 (1 pt). Implement the sentiment abstract data type, which represents a sentiment value that may or may not exist. The constructor make_sentiment takes either a numeric value within the interval -1 to 1, or None to indicate that the value does not exist. Implement the selectors has_sentiment and sentiment_value as well. You may use any representation you choose, but the rest of your program should not depend on this representation.

When you complete this problem, the doctests for make_sentiment and get_word_sentiment should pass. You can also call the print_sentiment function to print the sentiment values of all sentiment-carrying words in a line of text.

python3 trends.py -t make_sentiment
python3 trends.py -t get_word_sentiment
python3 trends.py -p computer science is my favorite!
python3 trends.py -p life without lambda: awful or awesome?


Problem 5 (1 pt). Implement analyze_tweet_sentiment, which takes a tweet (of the abstract data type) and returns a sentiment. Read the docstrings for get_word_sentiment and analyze_tweet_sentiment to understand how the two functions interact. Your implementation should not depend on the representation of a sentiment!.

When you complete this problem, the doctests for analyze_tweet_sentiment should pass.

python3 trends.py -t analyze_tweet_sentiment


Below is the solution for phase 1:

from data import word_sentiments, load_tweets
from datetime import datetime
from doctest import run_docstring_examples
from geo import us_states, geo_distance, make_position, longitude, latitude
from maps import draw_state, draw_name, draw_dot, wait, message
from string import ascii_letters
from ucb import main, trace, interact, log_current_line

# Phase 1: The Feelings in Tweets

def make_tweet(text, time, lat, lon):
"""Return a tweet, represented as a python dictionary.

text      -- A string; the text of the tweet, all in lowercase
time      -- A datetime object; the time that the tweet was posted
latitude  -- A number; the latitude of the tweet's location
longitude -- A number; the longitude of the tweet's location

>>> t = make_tweet("just ate lunch", datetime(2012, 9, 24, 13), 38, 74)
>>> tweet_words(t)
['just', 'ate', 'lunch']
>>> tweet_time(t)
datetime.datetime(2012, 9, 24, 13, 0)
>>> p = tweet_location(t)
>>> latitude(p)
38
"""
return {'text': text, 'time': time, 'latitude': lat, 'longitude': lon}

def tweet_words(tweet):
"""Return a list of the words in the text of a tweet."""
return extract_words(tweet['text'])

def tweet_time(tweet):
"""Return the datetime that represents when the tweet was posted."""
return tweet['time']

def tweet_location(tweet):
"""Return a position (see geo.py) that represents the tweet's location."""
return make_position(tweet['latitude'], tweet['longitude'])

def tweet_string(tweet):
"""Return a string representing the tweet."""
return '"{0}" @ {1}'.format(tweet['text'], tweet_location(tweet))

def extract_words(text):
"""Return the words in a tweet, not including punctuation.

>>> extract_words('anything  else.....not my job')
['anything', 'else', 'not', 'my', 'job']
>>> extract_words('i love my job. #winning')
['i', 'love', 'my', 'job', 'winning']
>>> extract_words('make justin # 1 by tweeting #vma #justinbieber :)')
['make', 'justin', 'by', 'tweeting', 'vma', 'justinbieber']
>>> extract_words("paperclips! they're so awesome, cool, & useful!")
['paperclips', 'they', 're', 'so', 'awesome', 'cool', 'useful']
"""
lst = []
current_index = 0
require_current_index_change = 0
for index, character in enumerate(text):
if character not in ascii_letters:
if not require_current_index_change:
require_current_index_change = 1
lst.append(text[current_index:index])
elif (character in ascii_letters) and (index == len(text) - 1):
lst.append(text[current_index:index+1])
break
else:
if require_current_index_change == 1:
current_index = index
require_current_index_change = 0
return lst

def make_sentiment(value):
"""Return a sentiment, which represents a value that may not exist.

>>> s = make_sentiment(0.2)
>>> t = make_sentiment(None)
>>> has_sentiment(s)
True
>>> has_sentiment(t)
False
>>> sentiment_value(s)
0.2
"""
assert value is None or (value >= -1 and value <= 1), 'Illegal value'
return value

def has_sentiment(s):
"""Return whether sentiment s has a value."""
if s == None:
return False
else:
return True

def sentiment_value(s):
"""Return the value of a sentiment s."""
assert has_sentiment(s), 'No sentiment value'
return s

def get_word_sentiment(word):
"""Return a sentiment representing the degree of positive or negative
feeling in the given word, if word is not in the sentiment dictionary.

>>> sentiment_value(get_word_sentiment('good'))
0.875
-0.625
>>> sentiment_value(get_word_sentiment('winning'))
0.5
>>> has_sentiment(get_word_sentiment('Berkeley'))
False
"""
return make_sentiment(word_sentiments.get(word, None))

def analyze_tweet_sentiment(tweet):
""" Return a sentiment representing the degree of positive or negative
sentiment in the given tweet, averaging over all the words in the tweet
that have a sentiment value.

If no words in the tweet have a sentiment value, return
make_sentiment(None).

>>> positive = make_tweet('i love my job. #winning', None, 0, 0)
>>> round(sentiment_value(analyze_tweet_sentiment(positive)), 5)
0.29167
>>> negative = make_tweet("Thinking, 'I hate my job'", None, 0, 0)
>>> sentiment_value(analyze_tweet_sentiment(negative))
-0.25
>>> no_sentiment = make_tweet("Go bears!", None, 0, 0)
>>> has_sentiment(analyze_tweet_sentiment(no_sentiment))
False
"""
average = make_sentiment(None)
words = tweet_words(tweet)
total_sentiment = 0
count_sentiment = 0
for word in words:
sentiment = get_word_sentiment(word)
if has_sentiment(sentiment):
total_sentiment += sentiment_value(sentiment)
count_sentiment += 1
if total_sentiment == 0:
return average
else:


As per instructions in phase 1, the solution was tested and looks fine.

Can I improve the solution code for the phase 1 assignment?

• Hi, your problem seems interesting. Is there any way to hilight parts of the code that needs to be reviewed ? (There are many things that could be done differently in the code you started with but I guess there is not much point in improving that code). May 19 '15 at 16:17
• @Josay there are 11 functions defined, one need to review all except tweet_string May 20 '15 at 9:21

The global design is a bit weird from my point of view but I'll comment on the code you've written.

In extract_words:

The code is properly formatted. A few remarks anyway :

• you don't need that many parenthesis.

• you don't need to check character in ascii_letters as it has to be true as this point.

• require_current_index_change looks like it should be a boolean. Just replace 1 by True, O by False and if require_current_index_change == 1: by if require_current_index_change:.

• Instead of having require_current_index_change to know whether you can use current_index or not, you could simply set current_index to None : it is easy to check and if you use the index anyway, you'll probably get an exception.

• You can get rid of the part comparing the index to the length and just handle it after the loop.

• current_index is probably not the best name as it let the reader think it corresponds to the index we are iterating over (aka index). It could be a good idea to convey the idea of beginning or starting index.

At the end, the code looks like :

def extract_words(text):
lst = []
starting_index = 0
for index, character in enumerate(text):
if character not in ascii_letters:
if starting_index is not None:
lst.append(text[starting_index:index])
starting_index = None
elif starting_index is None:
starting_index = index
if starting_index is not None:
lst.append(text[starting_index:])
return lst


Another idea would be to do things differently, by replacing unwanted characters by spaces and then to split on spaces.

In make_sentiment:

Instead of asserting, it could be an idea to raise a ValueError.

In has_sentiment:

You can simply : return s is not None.

Also, you should not compare to None using == but with is as per PEP8. You'll find various tools like pep8, pyflakes, etc to check your code and detect such things.

In analyze_tweet_sentiment:

Because non-zero integers value are considered True in boolean contexts, you can write :

if total_sentiment:
else:
return average


Which can be written :

    return total_sentiment / count_sentiment if total_sentiment else average


Also, average does not need to be defined that early, it could simply be :

    return total_sentiment / count_sentiment if total_sentiment else make_sentiment(None)


Then, I am wondering if you should be checking total_sentiment or count_sentiment. This corresponds to choose whether you can have a sentiment of value 0 (for instance if you have both positive and negative words) or if it corresponds to None. This is an open question and I do not have the answer.

Finally, a slightly different way to write this function would be to abuse list comprehension in order to be able to reuse builtin functions len and sum. For instance, we'd have something like :

def analyse(tweet):
sentiment_values = [sentiment_value(s) for s in (get_word_sentiment(w) for w in tweet_words(tweet)) if has_sentiment(s)]
return sum(sentiment_values)/ len(sentiment_values) if sentiment_values else make_sentiment(None)