2
\$\begingroup\$

I've wrote some simple code to load a JSON file from the Twitter Streaming API, parse the data, and create relationships between users based on who mentions or retweets who. My ultimate goal is to have a file that I can load into a network analysis program to analyze.

I wrote the code to first look for a retweet, and if present, create a relationship between the author and the user retweeted. Then look for any mentions, and if present, create a relationship between the author and any users mentioned. This part can have multiple relationships for a single tweet, as you can mention multiple people in one tweet. Then if neither a retweet or a mention is present, print the author and the tweet.

End goal of this is to simple parse a JSON file and return a CSV file that I can then analyze. I will not be loading this into any type of database.

import json
import sys
import time

def main():

    for line in sys.stdin:
        line = line.strip()

        data = []

        try:
            data.append(json.loads(line))
        except ValueError as detail:
            continue

        for tweet in data:
            ## deletes any rate limited data
            if tweet.has_key('limit'):
                pass

            else:
                tweet_time = time.strftime('%Y-%m-%d %H:%M:%S', time.strptime(tweet['created_at'],'%a %b %d %H:%M:%S +0000 %Y'))

                ## if the key 'retweeted_status' is in the dict, print
                if tweet.has_key('retweeted_status'):
                    print "\t".join([
                    tweet_time,
                    tweet['id_str'],
                    tweet['user']['screen_name'],
                    tweet['retweeted_status']['user']['screen_name'],
                    "RETWEET",
                    tweet['text']
                    ]).encode('utf8')

                ## if there is a mention in the dict, print
                elif 'entities' in tweet and len(tweet['entities']['user_mentions']) > 0:
                    for u2 in tweet['entities']['user_mentions']:
                        print "\t".join([
                        tweet_time,
                        tweet['id_str'],
                        tweet['user']['screen_name'],
                        u2['screen_name'],
                        "MENTION",
                        tweet['text']
                        ]).encode('utf8')

                ## if there is no retweet and no mention, print
                else:
                    print "\t".join([
                    tweet_time,
                    tweet['id_str'],
                    tweet['user']['screen_name'],
                    "\t",
                    "TWEET",
                    tweet['text']
                    ]).encode('utf8')

if __name__ == '__main__':
    main()
\$\endgroup\$
2
\$\begingroup\$

I would suggest to create following new classes:

  1. User, which stores information about user, like user_id, user_location
  2. Text, which stores the tweet text, the language of the tweet, may be the keywords
  3. Tweet, which contains objects of User ,Text and some other information about the tweet

I am assuming that you will be storing all this data into a database and the above mentioned structure (with some mild changes, based on your requirement) can be used to create a 3-NF database.

Another suggestion is "not" using stdin. Create a new file "Twitter.py" having function readTweet which will return tweet. This makes it easier to change the source of the tweet in the later part of the development. Suppose you want to directly take tweet from twitter API, then you just need to change the function

Your main function will look like following:

def main():

    for tweet in Twitter.getTweets():

        tweet_time = tweet.getTime() 

        if tweet.isRetweeted:
            print "do something 0"
        elif tweet.hasEntities and len(tweet.Usermentions) > 0:
            for mention in tweet.userMentions:
                print "do something 1"
        else:
            print "do something 2"
\$\endgroup\$
  • \$\begingroup\$ This looks like a good approach. However, I haven't learned about classes yet, so I will need to study up on this area. \$\endgroup\$ – CurtLH May 9 '14 at 16:27
  • \$\begingroup\$ The following link contains a nice tutorial for python classes: tutorialspoint.com/python/python_classes_objects.htm . WARNING: If you are Java programmer then you might have to leave many 'useless' OOPS concepts behind. \$\endgroup\$ – Pranav Raj May 9 '14 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.