8
\$\begingroup\$

I'm extremely new to Python, and the Twitter API, but I found an example online that walked me through the process. Now that I've been playing around with it for awhile, I've begun to push the limits of what my laptop can handle in terms of processing power with my current code. I was hoping someone here could check over it and make recommendations on optimizing it.

Goal: Take JSON output from the Twitter streaming API and print out specific fields to a CSV file. (The sysarg is used to pass the input and output filenames).

# import necessary modules
import json
import sys

# define a new variable for tweets
tweets=[]

# import tweets from JSON
for line in open(sys.argv[1]):
    try:
        tweets.append(json.loads(line))
    except:
        pass

# print the name of the file and number of tweets imported
print "File Imported:", str(sys.argv[1])
print "# Tweets Imported:", len(tweets)


# create a new variable for a single tweets
tweet=tweets[0]

# pull out various data from the tweets
tweet_id = [tweet['id'] for tweet in tweets]
tweet_text = [tweet['text'] for tweet in tweets]
tweet_time = [tweet['created_at'] for tweet in tweets]
tweet_author = [tweet['user']['screen_name'] for tweet in tweets]
tweet_author_id = [tweet['user']['id_str'] for tweet in tweets]
tweet_language = [tweet['lang'] for tweet in tweets]
tweet_geo = [tweet['geo'] for tweet in tweets]


#outputting to CSV
out = open(sys.argv[2], 'w')
print >> out, 'tweet_id, tweet_time, tweet_author, tweet_author_id,    tweet_language, tweet_geo, tweet_text'

rows = zip(tweet_id, tweet_time, tweet_author, tweet_author_id,    tweet_language, tweet_geo, tweet_text)

from csv import writer
csv = writer(out)

for row in rows:
    values = [(value.encode('utf8') if hasattr(value, 'encode') else value) for value in row]
    csv.writerow(values)

out.close()

# print name of exported file
print "File Exported:", str(sys.argv[2])
\$\endgroup\$

1 Answer 1

8
\$\begingroup\$
  • Put all import statements at the top.

  • You might leak a file descriptor, since you call open(sys.argv[1]) without closing it. (Whether a leak actually occurs depends on the garbage collector of your Python implementation.) Best practice is to use a with block, which automatically closes the resources when it terminates.

    with open(sys.argv[1]) as in_file, \
         open(sys.argv[2]) as out_file:
        …
    
  • It would be better to define your variables in the same order as they will appear in the CSV output.

  • Rather than creating an empty array and appending to it in a loop, use a list comprehension.

    tweets = [json.loads(line) for line in in_file]
    
  • You read all the tweets into an array of JSON objects, then slice the data "vertically" by attribute, then re-aggregate the data "horizontally". That's inefficient in terms of memory usage as well as cache locality.

  • Unless you have a good reason, just transform one line of input at a time. (A good reason might be that you want to produce no output file at all if an error occurs while processing any line.)

import json
import sys
from csv import writer

with open(sys.argv[1]) as in_file, \
     open(sys.argv[2], 'w') as out_file:
    print >> out_file, 'tweet_id, tweet_time, tweet_author, tweet_author_id,    tweet_language, tweet_geo, tweet_text'
    csv = writer(out_file)
    tweet_count = 0

    for line in in_file:
        tweet_count += 1
        tweet = json.loads(line)

        # Pull out various data from the tweets
        row = (
            tweet['id'],                    # tweet_id
            tweet['created_at'],            # tweet_time
            tweet['user']['screen_name'],   # tweet_author
            tweet['user']['id_str'],        # tweet_authod_id
            tweet['lang'],                  # tweet_language
            tweet['geo'],                   # tweet_geo
            tweet['text']                   # tweet_text
        )
        values = [(value.encode('utf8') if hasattr(value, 'encode') else value) for value in row]
        csv.writerow(values)

# print the name of the file and number of tweets imported
print "File Imported:", str(sys.argv[1])
print "# Tweets Imported:", tweet_count
print "File Exported:", str(sys.argv[2])
\$\endgroup\$
3
  • \$\begingroup\$ Thanks for the awesome review. I only had to make one very small change and it works perfectly. FYI, I changed the json import process to: try: tweet = json.loads(line) except: pass Thanks again for the review! \$\endgroup\$
    – CurtLH
    Mar 16, 2014 at 23:29
  • \$\begingroup\$ @Curtis Putting a try-except around every json.loads(line) may or may not be a good idea. That's your decision to make. However, swallowing exceptions silently is almost always a very bad idea. At the very least, print a complaint to sys.stderr. \$\endgroup\$ Mar 17, 2014 at 6:05
  • \$\begingroup\$ thanks again for all your help, it's all greatly appreciated it. I'm obviously very new to this, but the reason I thought I needed try-except is so that it won't blow up if it hits a incomplete record. Maybe this is incorrect, but it wouldn't run without it, so I guess I'll have to keep it in there until I learn of a better way. I'll look into sys.stderr and see what that can tell me. Thanks again for your help! \$\endgroup\$
    – CurtLH
    Mar 17, 2014 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.