Improving Twitter JSON parser for big data set

Question

I have the following fully working code that

1. imports JSON files,
2. parses the tweets contained in JSONs,
3. records them in a table in a data frame.

Considering that per run I currently analyze 1,400 JSONs (about 1.5Gb), the code takes quite some time to run. Please suggest if there is a plausible way to optimize the code in order to increase its speed. Thanks!

import os
import json
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import os

tweets = []

for dirs, subdirs, files in os.walk('/Users/mymac/Documents/Dir'):
    for file in files:
        if file.endswith('.json'):
            print(file)
            for line in open(file) :
                try:
                    tweet = json.loads(line)
                    tweets.append(tweet)
                except:
                    continue

tweet = tweets[0]

ids = [tweet['id_str'] for tweet in tweets if 'id_str' in tweet] 
text = [tweet['text'] for tweet in tweets if 'text' in tweet]
lang = [tweet['lang'] for tweet in tweets if 'lang' in tweet]
geo = [tweet['geo'] for tweet in tweets if 'geo' in tweet]                    
place = [tweet['place'] for tweet in tweets if 'place' in tweet]

df=pd.DataFrame({'Ids':pd.Index(ids),
                 'Text':pd.Index(text),
                 'Lang':pd.Index(lang),
                 'Geo':pd.Index(geo),
                 'Place':pd.Index(place)})
df

Answer 1

Just a few quick considerations:

• You have import os twice
• You are not using matplotlib and numpy, so the imports can go
• The line tweet = tweets[0] is useless
• You're not closing the files you open, you should use the with keyword

Two optimizations:

• I'd remove the print(file). This is probably single best optimization you can do
• You're already looping once, why do you loop another five times?

What about something like this (not tested!):

from collections import defaultdict

elements_keys = ['ids', 'text', 'lang', 'geo', 'place']
elements = defaultdict(list)

for dirs, subdirs, files in os.walk('/Users/mymac/Documents/Dir'):
    for file in files:
        if file.endswith('.json'):
            with open(file, 'r') as input_file:
                for line in input_file:
                    try:
                        tweet = json.loads(line)
                        for key in elements_keys:
                            elements[key].append(tweet[key])
                    except:
                        continue

df=pd.DataFrame({'Ids': pd.Index(elements['id']),
                 'Text': pd.Index(elements['text']),
                 'Lang': pd.Index(elements['lang']),
                 'Geo': pd.Index(elements['geo']),
                 'Place': pd.Index(elements['place'])})
df