7
\$\begingroup\$

Here is a Python script that streams Twitter either with a keyword or within a bounded geographic area. Errors are emailed to my Gmail and the tweets are stored locally in MongoDB. Please let me know if any improvements can be made.

import tweepy
import sys
import pymongo
import smtplib
import sys

#The following functions send specific error codes to me then end the stream.

def email_me_error420():
    server = smtplib.SMTP('smtp.gmail.com', 587)
    server.starttls()
    server.login("xxx@gmail.com", "xxx")

    msg = "The twitter stream received a 420 error and disconnected! You are being rate limited by Twitter."
    server.sendmail("xxx@gmail.com", "xxx@gmail.com", msg)
    server.quit()
    sys.exit()

def email_me_error500():
    server = smtplib.SMTP('smtp.gmail.com', 587)
    server.starttls()
    server.login("xxx@gmail.com", "xxx")

    msg = "The twitter stream received a 500 error and disconnected! Someone broke twitter...! Maybe Doomsday?"
    server.sendmail("xxx@gmail.com", "xxx@gmail.com", msg)
    server.quit()
    sys.exit()

def email_me_error502():
    server = smtplib.SMTP('smtp.gmail.com', 587)
    server.starttls()
    server.login("xxx@gmail.com", "xxx")

    msg = "The twitter stream received a 502 error and disconnected!"
    server.sendmail("xxx@gmail.com", "xxx@gmail.com", msg)
    server.quit()
    sys.exit()

def email_me_error503():
    server = smtplib.SMTP('smtp.gmail.com', 587)
    server.starttls()
    server.login("xxx@gmail.com", "xxx")

    msg = "The twitter stream received a 503 error and disconnected!"
    server.sendmail("xxx@gmail.com", "xxx@gmail.com", msg)
    server.quit()
    sys.exit()

def email_me_error504():
    server = smtplib.SMTP('smtp.gmail.com', 587)
    server.starttls()
    server.login("xxx@gmail.com", "xxx")

    msg = "The twitter stream received a 504 error and disconnected!"
    server.sendmail("xxx@gmail.com", "xxx@gmail.com", msg)
    server.quit()
    sys.exit()

access_token = "1111-1111"
access_token_secret = "1111"
consumer_key = "1111"
consumer_secret = "1111"

#set up authorization and api access

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)

#stream listener is below

class CustomStreamListener(tweepy.StreamListener):
    def __init__(self, api):
        self.api = api
        super(tweepy.StreamListener, self).__init__()

        self.db = pymongo.MongoClient().WorldTweetsGeo

    def on_error(self, status_code):
        if status_code == 420:
            email_me_error420()
            print (status_code)
            sys.exit()
            return False
        elif status_code == 500:
            email_me_error500()
            print (status_code)
            sys.exit()
            return False
        elif status_code == 502:
            email_me_error502()
            print (status_code)
            sys.exit()
            return False
        elif status_code == 503:
            email_me_error503()
            print (status_code)
            sys.exit()
            return False
        elif status_code == 504:
            email_me_error504()
            print (status_code)
            sys.exit()
            return False
        else:
            print (status_code)

    def on_status(self, status):

        data ={}

        data['TweetText'] = status.text
        data['Name'] = status.user.name
        data['UserDesc'] = status.user.description
        data['Coordinates'] = status.geo
        data['GeoEnabled'] = status.user.geo_enabled
        data['UserLocation'] = status.user.location
        data['TweetRetweetCt'] = status.retweet_count
        data['TweetFavoriteCt'] = status.favorite_count
        data['TweetSource'] = status.source
        data['TweetCreated'] = status.created_at
        data['UserID'] = status.user.id_str
        data['UserScreen'] = status.user.screen_name
        data['UserName'] = status.user.name
        data['UserFollowerCt'] = status.user.followers_count
        data['UserTimezone'] = status.user.time_zone
        data['UtcOffset'] = status.user.utc_offset
        data['FollowersCount'] = status.user.followers_count
        data['FriendsCount'] = status.user.friends_count
        data['StatusesCount'] = status.user.statuses_count
        data['AccountCreated'] = status.user.created_at
        data['Language'] = status.lang
        data['PlaceID'] = status.place.country_code
        data['InReplytoID'] = status.in_reply_to_user_id_str
        data['InReplytoStatusID'] = status.in_reply_to_status_id_str

        self.db.Tweets.insert(data)

location = (111, 111, 111, 111)

def launch_twitter_stream():
    while True:
        try:
            sapi = tweepy.streaming.Stream(auth, CustomStreamListener(api))
            sapi.filter(locations=(location))
            #sapi.filter(track=["keyword"])
        except KeyboardInterrupt:
            sys.exit()
        except: 
            continue

launch_twitter_stream()
\$\endgroup\$
  • 2
    \$\begingroup\$ You have so much repeated code when handling the errors, personally I would make a function for the server login and pass it as an argument. Same thing with your if elif else. The only thing that really changes is the message itself. \$\endgroup\$ – Navidad20 Jan 27 '17 at 14:23
12
\$\begingroup\$

You can avoid almost all of your redundancy by writing a function that accepts the status code as parameter.

I defined a collections.defaultdict dictionary for the two status codes where there is some additional error message. For all other keys, this dictionary will return the default value (a str() == "" in this case).

I also put a if __name__ == "__main__": guard around your main code to allow importing the functions from this script in another script without executing the listener.

I would also put the api setup into its own function.

In addition, the usual structure of python scripts is:

#!/usr/bin/env python
"""Module description docstring"""

import function from module

CONSTANT = ...

class Definition:
    """Class docstring"""
    ...


def function():
    """Function docstring"""
    ...


def main():
    """Main function"""
    ...


if __name__ == '__main__':
    main()

Therefore I re-ordered your code somewhat.

import tweepy
import sys
import pymongo
import smtplib
import sys

from collections import defaultdict

SPECIFIC_MSG = defaultdict(str, {
    420: " You are being rate limited by Twitter.",
    500: " Someone broke twitter...! Maybe Doomsday?"})


class CustomStreamListener(tweepy.StreamListener):
    def __init__(self, api):
        self.api = api
        super(tweepy.StreamListener, self).__init__()

        self.db = pymongo.MongoClient().WorldTweetsGeo

    def on_error(self, status_code):
        print(status_code)
        if status_code in {420, 500, 502, 503, 504}:
            email_me_error(status_code)
            sys.exit()

    def on_status(self, status):

        data ={}

        data['TweetText'] = status.text
        data['Name'] = status.user.name
        data['UserDesc'] = status.user.description
        data['Coordinates'] = status.geo
        data['GeoEnabled'] = status.user.geo_enabled
        data['UserLocation'] = status.user.location
        data['TweetRetweetCt'] = status.retweet_count
        data['TweetFavoriteCt'] = status.favorite_count
        data['TweetSource'] = status.source
        data['TweetCreated'] = status.created_at
        data['UserID'] = status.user.id_str
        data['UserScreen'] = status.user.screen_name
        data['UserName'] = status.user.name
        data['UserFollowerCt'] = status.user.followers_count
        data['UserTimezone'] = status.user.time_zone
        data['UtcOffset'] = status.user.utc_offset
        data['FollowersCount'] = status.user.followers_count
        data['FriendsCount'] = status.user.friends_count
        data['StatusesCount'] = status.user.statuses_count
        data['AccountCreated'] = status.user.created_at
        data['Language'] = status.lang
        data['PlaceID'] = status.place.country_code
        data['InReplytoID'] = status.in_reply_to_user_id_str
        data['InReplytoStatusID'] = status.in_reply_to_status_id_str

        self.db.Tweets.insert(data)

def email_me_error(status_code):
    """Sends specific error codes to me."""

    server = smtplib.SMTP('smtp.gmail.com', 587)
    server.starttls()
    server.login("xxx@gmail.com", "xxx")

    msg = "The twitter stream received a {} error and disconnected!{}".format(status_code, SPECIFIC_MSG[status_code])
    server.sendmail("xxx@gmail.com", "xxx@gmail.com", msg)
    server.quit()


def setup_api():
    access_token = "1111-1111"
    access_token_secret = "1111"
    consumer_key = "1111"
    consumer_secret = "1111"

    #set up authorization and api access
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)
    return auth, api


def launch_twitter_stream(auth, api, **filter_kwargs):
    while True:
        try:
            sapi = tweepy.streaming.Stream(auth, CustomStreamListener(api))
            sapi.filter(**filter_kwargs)
        except Exception: 
            continue

if __name__ == "__main__":
    location = (111, 111, 111, 111)
    auth, api = setup_api()
    launch_twitter_stream(auth, api, location=location)
    # launch_twitter_stream(auth, api, track=["keyword"])
\$\endgroup\$
  • \$\begingroup\$ Nice answer. In SPECIFIC_MSG you forgot a ) at the end. It might be due to the weird indentation for that defaultdict which makes this harder to spot. ^^ I'd personally go with this \$\endgroup\$ – Grajdeanu Alex. Jan 27 '17 at 22:54
  • \$\begingroup\$ @Dex'ter Thanks, fixed! Yeah I'm never sure how to properly indent that. Normally my editor does it for me, but I wrote this code solely on the browser... \$\endgroup\$ – Graipher Jan 27 '17 at 22:57
  • \$\begingroup\$ Thanks! That helps a lot. I really appreciate the help. Marked as answered. \$\endgroup\$ – walk_for_life Jan 29 '17 at 0:36
  • \$\begingroup\$ Could you offer some insight into how to drop the location argument? I sometimes run the stream without a geocoded area, filtering on a keyword instead. You can only do one or the other at any one time- location or keyword. I changed the arguments to not need location, but it loops and I get a 420 from twitter. Any ideas? Thanks again. \$\endgroup\$ – walk_for_life Jan 29 '17 at 2:27
  • \$\begingroup\$ @walk_for_life You could add a filter parameter and then pass a dictionary there (e.g. {'location': (111,111,111,111)} or {'track': ["keyword"]}). Then do sapi.filter(**filter). But I have no idea about the Twitter API, so no clue if you are using it wrong... \$\endgroup\$ – Graipher Jan 29 '17 at 10:42
3
\$\begingroup\$

There are several major violations in the code:

  • DRY principle - the email_me_error* functions are almost the same except the error messages
  • the scope and the things your functions do are not defined properly - for example, the email_me_error* functions also do things they are not supposed to do - connecting to the email server (should be extracted into a separate function) and exiting the program
  • the SMTP and other email settings should be extracted to a higher level - into constants or into a separate "configuration" module
  • handling bare exceptions with except: or except Exception: is considered a bad practice

To improve the email error reporting, I would actually use a logger - for example, there is the SMTPHandler logging handler in the built-in logging library.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.