12
\$\begingroup\$

I want to have someone reviewing and helping me get some good ideas in how to improve my current project.

All information about the project is in the following github link.

I would also appreciate ideas on how to plot in a more efficient way.

Example of the program with 3 minutes in action

Code:

import tweepy, json
from tweepy.api import API
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
from textblob import TextBlob
import re

""" Variables 
================"""
searches  = ['trump']    # max 250 input searches in a list of strings
maxTweets = 100000         # max number of tweets to analyse.

# Get the Authentication keys from another file. (Easier to push to git)
with open("/Users/se/Documents/Lek med Python/pyCharmProjects/dataScienceFromScratch/src/twitterKeys.txt","r") as file:
    CONSUMER_KEY        = file.readline().rstrip()
    CONSUMER_SECRET     = file.readline().rstrip()
    ACCESS_TOKEN        = file.readline().rstrip()
    ACCESS_TOKEN_SECRET = file.readline().rstrip()

# Twitter API needs to be validated
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

# The class takes in a stream of tweets and filter out some specific properties.
# It is a live streaming so it will be kind of slow and in this purpose i filter
# the Coordinates which only around 3-4% of all tweets have.
class Stream2Screen(tweepy.StreamListener):
    def __init__(self, api=None):
        self.api = api or API()
        self.counter = 0            # Counts amount of tweets already got



    # This is the method that recieves the tweet in the name of data.
    def on_data(self, data):
        datadict = json.loads(data)     # easier to handle in a json file

        # Checks so that the coordinate is in the tweet
        if (datadict.get('place') != None):
            coordinate = calcCenterOfPolygone(datadict['place']['bounding_box']['coordinates'][0])
            tweet = cleanTweets(datadict['text'])
            sentiment = sentimentAnalysis(tweet)
            plotOnMap(coordinate, sentiment.polarity)
            print(tweet)
            print(sentiment, "\n")
            self.counter += 1
            if(self.counter % 100 == 0):
                print(self.counter)

        # if we sent a False as return value we close the "pipeline" of streams
        if self.counter < maxTweets:
            return True
        else:
            return False

    def on_error(self, status_code):
        print(status_code)
        return True

    # Trial to get sentimental value of a tweet using TextBlob
def sentimentAnalysis(tweet):
    tweet = TextBlob(cleanTweets(tweet))
    return tweet.sentiment

def cleanTweets(tweet):
    return ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)", " ", tweet).split())


# Inputs 4 Coordinates in a list and calc the mid point.
def calcCenterOfPolygone(coordinates):
    return [sum([x[0] for x in coordinates]) / 4,
            sum([x[1] for x in coordinates]) / 4]

# Plots the coordinate on the map
def plotOnMap(centerPoint, polarity):
    plt.ion()
    color = 'black'
    if (polarity < 0): color='r'
    elif (polarity > 0): color='g'
    m.plot(centerPoint[0], centerPoint[1], marker='o', markersize=5, color=color)
    draw_map(m)
    plt.pause(0.000001)

# draws the map the first time
def draw_map(m, scale=0.2):
    m.shadedrelief(scale=scale)


fig = plt.figure(figsize=(12, 6))
m = Basemap(projection='cyl', resolution=None,
            llcrnrlat=-90, urcrnrlat=90,
            llcrnrlon=-180, urcrnrlon=180,)

# Starts the twitter feed and searches for the keyword
stream = tweepy.streaming.Stream(auth, Stream2Screen())

while True:
    try:
        stream.filter(track=searches)
    except:
        print("error"*10)
        continue

draw_map(m)
plt.show()
plt.draw()
\$\endgroup\$
  • 1
    \$\begingroup\$ Does the code currently work as good as expected? Does it reach the required accuracy? \$\endgroup\$ – Mast Feb 6 '18 at 10:11
  • \$\begingroup\$ I can say its an working progress. It currently work as it intends to. and a loot of information is on my page about precision. But i'm sort of new to the process so cant be sure its a good optimisation! \$\endgroup\$ – Edvardsson Feb 6 '18 at 12:24
  • 1
    \$\begingroup\$ @Mast I've edited out the off-topic portion of the question. \$\endgroup\$ – 200_success Feb 7 '18 at 18:38
  • \$\begingroup\$ @IEatBagels It's plotting the sentimental values of tweets on a chart, as illustrated by the example. If you need more information, please be specific. \$\endgroup\$ – Mast Jul 29 at 9:31
2
\$\begingroup\$

Alright time to give this a shot!

Code style

I won't go in deep details, but the PEP8 standard is that methods should use snake_case not camelCase. You can do whatever you want with this information.

Comments

In my opinion, there are way too many comments in your code. Most of them can be removed with some refactoring. I'll try to target the different reasons why some things shouldn't be commented and the associated refactors for them.

There's one for which I don't have a refactor, because it's an old-school method of doing this that, I believe, shouldn't be done anymore :

""" Variables 
================"""

Don't do this. People that will read your code will figure out that variables are following. If you change your code over time and keep this comment it's probable that this comment will become obsolete.

There's also the case of this comment : # if we sent a False as return value we close the "pipeline" of streams which just brings a load of confusion in the code. This comment probably doesn't need to exist as when we will see where the method on_data is called, we'll figure it out ourself.

Refactor 1 : Replace comments with better variable names.

maxTweets = 100000 # max number of tweets to analyse.

If you feel the need to put this comment, it's probably because your variable isn't well named. If you named your variable maxTweetsToAnalayse you wouldn't need the comment.

self.counter = 0            # Counts amount of tweets already got

Same here, you should rename your variable, maybe loadedTweetsCount or something like that?

# Checks so that the coordinate is in the tweet
if (datadict.get('place') != None):

Could be replaced by :

coordinateIsInTweet = datadict.get('place') != None
if coordinateIsInTweet:

Refactor 2 : Write validation rules for comments stating rules.

# max 250 input searches in a list of strings

That's something your code should be aware of. Instead of this comment, why not validate it in the code?

if len(searches) > 250:
   raise ValueError("Exceeded maximum searches inputs")

Refactor 3 : Write a method for a comment explaining a bunch of code lines.

# Get the Authentication keys from another file. (Easier to push to git)
# Twitter API needs to be validated

Instead of having this comments, why not do this :

def getTwitterAuthToken():
    with open("/Users/se/Documents/Lek med Python/pyCharmProjects/dataScienceFromScratch/src/twitterKeys.txt","r") as file:
        CONSUMER_KEY        = file.readline().rstrip()
        CONSUMER_SECRET     = file.readline().rstrip()
        ACCESS_TOKEN        = file.readline().rstrip()
        ACCESS_TOKEN_SECRET = file.readline().rstrip()

    auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
    auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

    return auth

The first comment was unnecessary, there are plenty of reasons why a configuration file is better than having hardcoded credentials in your code and your code is clear enough that well... the comment isn't necessary.

Refactor 4 : Use documentation instead of single line comments

# The class takes in a stream of tweets and filter out some specific properties.
# It is a live streaming so it will be kind of slow and in this purpose i filter
# the Coordinates which only around 3-4% of all tweets have.

This explains what the class does, so it should be inside the class. You can create class documentation by doing this (More info here):

class Stream2Screen(tweepy.StreamListener):

    """The class takes in a stream of tweets and filter out some specific properties.
    It is a live streaming so it will be kind of slow and in this purpose i filter
    the Coordinates which only around 3-4% of all tweets have."""

Same can be done for methods :

# This is the method that receives the tweet in the name of data.
def on_data(self, data):

Becomes

def on_data(self, data):
    """This is the method that receives the tweet in the name of data."""

That's about it for comments, I think this was the biggest "weakness" in the code and those refactorings will help make the code much better.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.