I'm extremely new to Python, and the Twitter API, but I found an example online that walked me through the process. Now that I've been playing around with it for awhile, I've begun to push the limits of what my laptop can handle in terms of processing power with my current code. I was hoping someone here could check over it and make recommendations on optimizing it.
Goal: Take JSON output from the Twitter streaming API and print out specific fields to a CSV file. (The sysarg is used to pass the input and output filenames).
# import necessary modules
import json
import sys
# define a new variable for tweets
tweets=[]
# import tweets from JSON
for line in open(sys.argv[1]):
try:
tweets.append(json.loads(line))
except:
pass
# print the name of the file and number of tweets imported
print "File Imported:", str(sys.argv[1])
print "# Tweets Imported:", len(tweets)
# create a new variable for a single tweets
tweet=tweets[0]
# pull out various data from the tweets
tweet_id = [tweet['id'] for tweet in tweets]
tweet_text = [tweet['text'] for tweet in tweets]
tweet_time = [tweet['created_at'] for tweet in tweets]
tweet_author = [tweet['user']['screen_name'] for tweet in tweets]
tweet_author_id = [tweet['user']['id_str'] for tweet in tweets]
tweet_language = [tweet['lang'] for tweet in tweets]
tweet_geo = [tweet['geo'] for tweet in tweets]
#outputting to CSV
out = open(sys.argv[2], 'w')
print >> out, 'tweet_id, tweet_time, tweet_author, tweet_author_id, tweet_language, tweet_geo, tweet_text'
rows = zip(tweet_id, tweet_time, tweet_author, tweet_author_id, tweet_language, tweet_geo, tweet_text)
from csv import writer
csv = writer(out)
for row in rows:
values = [(value.encode('utf8') if hasattr(value, 'encode') else value) for value in row]
csv.writerow(values)
out.close()
# print name of exported file
print "File Exported:", str(sys.argv[2])