5
\$\begingroup\$

Below assignment is taken from here.

Introduction

In this project, you will develop a geographic visualization of twitter data across the USA. You will need to use dictionaries, lists, and data abstraction techniques to create a modular program. Below is phase 1 of this project.

Phase 1: The Feelings in Tweets

In this phase, you will create an abstract data type for tweets, split the text of a tweet into words, and calculate the amount of positive or negative feeling in a tweet.

Tweets

First, you will implement an abstract data type for Tweets. The constructor make_tweet is defined at the top of trends.py. make_tweet returns a python dictionary with the following entries:

  text:      a string, the text of the tweet, all in lowercase
  time:      a datetime object, when the tweet was posted
  latitude:  a floating-point number, the latitude of the tweet's location
  longitude: a floating-point number, the longitude of the tweet's location

Problem 1 (1 pt). Implement the tweet_words and tweet_time selectors. Call extract_words to list the words in the text of a tweet.

Problem 2 (1 pt). Implement the tweet_location selector, which returns a position. Positions are another abstract data type, defined at the top of geo.py. Make sure that you understand how to manipulate positions; they play an important role in this project.

When you complete problems 1 and 2, the doctest for make_tweet should pass.

python3 trends.py -t make_tweet

Problem 3 (1 pt). Improve the extract_words function as follows: Assume that a word is any consecutive substring of text that consists only of ASCII letters. The string ascii_letters in the string module contains all letters in the ASCII character set. The extract_words function should list all such words in order and nothing else.

When you complete this problem, the doctest for extract_words should pass.

python3 trends.py -t extract_words

Problem 4 (1 pt). Implement the sentiment abstract data type, which represents a sentiment value that may or may not exist. The constructor make_sentiment takes either a numeric value within the interval -1 to 1, or None to indicate that the value does not exist. Implement the selectors has_sentiment and sentiment_value as well. You may use any representation you choose, but the rest of your program should not depend on this representation.

When you complete this problem, the doctests for make_sentiment and get_word_sentiment should pass. You can also call the print_sentiment function to print the sentiment values of all sentiment-carrying words in a line of text.

python3 trends.py -t make_sentiment
python3 trends.py -t get_word_sentiment
python3 trends.py -p computer science is my favorite!
python3 trends.py -p life without lambda: awful or awesome?

Problem 5 (1 pt). Implement analyze_tweet_sentiment, which takes a tweet (of the abstract data type) and returns a sentiment. Read the docstrings for get_word_sentiment and analyze_tweet_sentiment to understand how the two functions interact. Your implementation should not depend on the representation of a sentiment!.

When you complete this problem, the doctests for analyze_tweet_sentiment should pass.

python3 trends.py -t analyze_tweet_sentiment

Below is the solution for phase 1:

from data import word_sentiments, load_tweets
from datetime import datetime
from doctest import run_docstring_examples
from geo import us_states, geo_distance, make_position, longitude, latitude
from maps import draw_state, draw_name, draw_dot, wait, message
from string import ascii_letters
from ucb import main, trace, interact, log_current_line


# Phase 1: The Feelings in Tweets

def make_tweet(text, time, lat, lon):
    """Return a tweet, represented as a python dictionary.

    text      -- A string; the text of the tweet, all in lowercase
    time      -- A datetime object; the time that the tweet was posted
    latitude  -- A number; the latitude of the tweet's location
    longitude -- A number; the longitude of the tweet's location

    >>> t = make_tweet("just ate lunch", datetime(2012, 9, 24, 13), 38, 74)
    >>> tweet_words(t)
    ['just', 'ate', 'lunch']
    >>> tweet_time(t)
    datetime.datetime(2012, 9, 24, 13, 0)
    >>> p = tweet_location(t)
    >>> latitude(p)
    38
    """
    return {'text': text, 'time': time, 'latitude': lat, 'longitude': lon}

def tweet_words(tweet):
    """Return a list of the words in the text of a tweet."""
    return extract_words(tweet['text'])

def tweet_time(tweet):
    """Return the datetime that represents when the tweet was posted."""
    return tweet['time']

def tweet_location(tweet):
    """Return a position (see geo.py) that represents the tweet's location."""
    return make_position(tweet['latitude'], tweet['longitude'])

def tweet_string(tweet):
    """Return a string representing the tweet."""
    return '"{0}" @ {1}'.format(tweet['text'], tweet_location(tweet))

def extract_words(text):
    """Return the words in a tweet, not including punctuation.

    >>> extract_words('anything  else.....not my job')
    ['anything', 'else', 'not', 'my', 'job']
    >>> extract_words('i love my job. #winning')
    ['i', 'love', 'my', 'job', 'winning']
    >>> extract_words('make justin # 1 by tweeting #vma #justinbieber :)')
    ['make', 'justin', 'by', 'tweeting', 'vma', 'justinbieber']
    >>> extract_words("paperclips! they're so awesome, cool, & useful!")
    ['paperclips', 'they', 're', 'so', 'awesome', 'cool', 'useful']
    """
    lst = []
    current_index = 0
    require_current_index_change = 0
    for index, character in enumerate(text): 
        if character not in ascii_letters:
            if not require_current_index_change:
                require_current_index_change = 1
                lst.append(text[current_index:index])
        elif (character in ascii_letters) and (index == len(text) - 1):
            lst.append(text[current_index:index+1])
            break
        else:
            if require_current_index_change == 1:
                current_index = index
                require_current_index_change = 0                
    return lst

def make_sentiment(value):
    """Return a sentiment, which represents a value that may not exist.

    >>> s = make_sentiment(0.2)
    >>> t = make_sentiment(None)
    >>> has_sentiment(s)
    True
    >>> has_sentiment(t)
    False
    >>> sentiment_value(s)
    0.2
    """
    assert value is None or (value >= -1 and value <= 1), 'Illegal value'
    return value

def has_sentiment(s):
    """Return whether sentiment s has a value."""
    if s == None:
        return False    
    else:
        return True

def sentiment_value(s):
    """Return the value of a sentiment s."""
    assert has_sentiment(s), 'No sentiment value'
    return s

def get_word_sentiment(word):
    """Return a sentiment representing the degree of positive or negative
    feeling in the given word, if word is not in the sentiment dictionary.

    >>> sentiment_value(get_word_sentiment('good'))
    0.875
    >>> sentiment_value(get_word_sentiment('bad'))
    -0.625
    >>> sentiment_value(get_word_sentiment('winning'))
    0.5
    >>> has_sentiment(get_word_sentiment('Berkeley'))
    False
    """
    return make_sentiment(word_sentiments.get(word, None))

def analyze_tweet_sentiment(tweet):
    """ Return a sentiment representing the degree of positive or negative
    sentiment in the given tweet, averaging over all the words in the tweet
    that have a sentiment value.

    If no words in the tweet have a sentiment value, return
    make_sentiment(None).

    >>> positive = make_tweet('i love my job. #winning', None, 0, 0)
    >>> round(sentiment_value(analyze_tweet_sentiment(positive)), 5)
    0.29167
    >>> negative = make_tweet("Thinking, 'I hate my job'", None, 0, 0)
    >>> sentiment_value(analyze_tweet_sentiment(negative))
    -0.25
    >>> no_sentiment = make_tweet("Go bears!", None, 0, 0)
    >>> has_sentiment(analyze_tweet_sentiment(no_sentiment))
    False
    """
    average = make_sentiment(None)
    words = tweet_words(tweet)
    total_sentiment = 0
    count_sentiment = 0
    for word in words:
        sentiment = get_word_sentiment(word)    
        if has_sentiment(sentiment):
            total_sentiment += sentiment_value(sentiment)
            count_sentiment += 1            
    if total_sentiment == 0:
        return average
    else:
        return total_sentiment / count_sentiment            

As per instructions in phase 1, the solution was tested and looks fine.

Can I improve the solution code for the phase 1 assignment?

\$\endgroup\$
  • \$\begingroup\$ Hi, your problem seems interesting. Is there any way to hilight parts of the code that needs to be reviewed ? (There are many things that could be done differently in the code you started with but I guess there is not much point in improving that code). \$\endgroup\$ – SylvainD May 19 '15 at 16:17
  • \$\begingroup\$ @Josay there are 11 functions defined, one need to review all except tweet_string \$\endgroup\$ – overexchange May 20 '15 at 9:21
2
\$\begingroup\$

The global design is a bit weird from my point of view but I'll comment on the code you've written.

In extract_words:

The code is properly formatted. A few remarks anyway :

  • you don't need that many parenthesis.

  • you don't need to check character in ascii_letters as it has to be true as this point.

  • require_current_index_change looks like it should be a boolean. Just replace 1 by True, O by False and if require_current_index_change == 1: by if require_current_index_change:.

  • Instead of having require_current_index_change to know whether you can use current_index or not, you could simply set current_index to None : it is easy to check and if you use the index anyway, you'll probably get an exception.

  • You can get rid of the part comparing the index to the length and just handle it after the loop.

  • current_index is probably not the best name as it let the reader think it corresponds to the index we are iterating over (aka index). It could be a good idea to convey the idea of beginning or starting index.

At the end, the code looks like :

def extract_words(text):
    lst = []
    starting_index = 0
    for index, character in enumerate(text):
        if character not in ascii_letters:
            if starting_index is not None:
                lst.append(text[starting_index:index])
            starting_index = None
        elif starting_index is None:
            starting_index = index
    if starting_index is not None:
        lst.append(text[starting_index:])
    return lst

Another idea would be to do things differently, by replacing unwanted characters by spaces and then to split on spaces.

In make_sentiment:

Instead of asserting, it could be an idea to raise a ValueError.

In has_sentiment:

You can simply : return s is not None.

Also, you should not compare to None using == but with is as per PEP8. You'll find various tools like pep8, pyflakes, etc to check your code and detect such things.

In analyze_tweet_sentiment:

Because non-zero integers value are considered True in boolean contexts, you can write :

if total_sentiment:
    return total_sentiment / count_sentiment    
else:
    return average

Which can be written :

    return total_sentiment / count_sentiment if total_sentiment else average

Also, average does not need to be defined that early, it could simply be :

    return total_sentiment / count_sentiment if total_sentiment else make_sentiment(None)

Then, I am wondering if you should be checking total_sentiment or count_sentiment. This corresponds to choose whether you can have a sentiment of value 0 (for instance if you have both positive and negative words) or if it corresponds to None. This is an open question and I do not have the answer.

Finally, a slightly different way to write this function would be to abuse list comprehension in order to be able to reuse builtin functions len and sum. For instance, we'd have something like :

def analyse(tweet):
    sentiment_values = [sentiment_value(s) for s in (get_word_sentiment(w) for w in tweet_words(tweet)) if has_sentiment(s)]
    return sum(sentiment_values)/ len(sentiment_values) if sentiment_values else make_sentiment(None)
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.