# Print Tweets/Mentions/Retweets

I've wrote some simple code to load a JSON file from the Twitter Streaming API, parse the data, and create relationships between users based on who mentions or retweets who. My ultimate goal is to have a file that I can load into a network analysis program to analyze.

I wrote the code to first look for a retweet, and if present, create a relationship between the author and the user retweeted. Then look for any mentions, and if present, create a relationship between the author and any users mentioned. This part can have multiple relationships for a single tweet, as you can mention multiple people in one tweet. Then if neither a retweet or a mention is present, print the author and the tweet.

End goal of this is to simple parse a JSON file and return a CSV file that I can then analyze. I will not be loading this into any type of database.

import json
import sys
import time

def main():

for line in sys.stdin:
line = line.strip()

data = []

try:
except ValueError as detail:
continue

for tweet in data:
## deletes any rate limited data
if tweet.has_key('limit'):
pass

else:
tweet_time = time.strftime('%Y-%m-%d %H:%M:%S', time.strptime(tweet['created_at'],'%a %b %d %H:%M:%S +0000 %Y'))

## if the key 'retweeted_status' is in the dict, print
if tweet.has_key('retweeted_status'):
print "\t".join([
tweet_time,
tweet['id_str'],
tweet['user']['screen_name'],
tweet['retweeted_status']['user']['screen_name'],
"RETWEET",
tweet['text']
]).encode('utf8')

## if there is a mention in the dict, print
elif 'entities' in tweet and len(tweet['entities']['user_mentions']) > 0:
for u2 in tweet['entities']['user_mentions']:
print "\t".join([
tweet_time,
tweet['id_str'],
tweet['user']['screen_name'],
u2['screen_name'],
"MENTION",
tweet['text']
]).encode('utf8')

## if there is no retweet and no mention, print
else:
print "\t".join([
tweet_time,
tweet['id_str'],
tweet['user']['screen_name'],
"\t",
"TWEET",
tweet['text']
]).encode('utf8')

if __name__ == '__main__':
main()


I would suggest to create following new classes:

1. User, which stores information about user, like user_id, user_location
2. Text, which stores the tweet text, the language of the tweet, may be the keywords
3. Tweet, which contains objects of User ,Text and some other information about the tweet

I am assuming that you will be storing all this data into a database and the above mentioned structure (with some mild changes, based on your requirement) can be used to create a 3-NF database.

Another suggestion is "not" using stdin. Create a new file "Twitter.py" having function readTweet which will return tweet. This makes it easier to change the source of the tweet in the later part of the development. Suppose you want to directly take tweet from twitter API, then you just need to change the function

Your main function will look like following:

def main():