I created a Pandas dataframe from a MongoDB query.
c = db.runs.find().limit(limit) df = pd.DataFrame(list(c))
Right now one column of the dataframe corresponds to a document nested within the original MongoDB document, now typed as a dictionary. The dictionary is in the
I would like to extract some of the dictionary's values to make new columns of the data frame. Is there a general way to do this? If not, what is the best brute force method specific to my data?
Here is my brute force approach, and it's horrendous. Even for a dataframe of only 200 rows, this is taking several minutes:
run_info = df['run_info'] # want to extra values from dictionary stored in column df['run_info'] # some values are within dictionaries nested within top dictionary # these are 'author' and 'weather' for i in range(len(df['run'])): g = run_info[i] if g.get('weather'): for name in weather_list: if g.get('weather').get(name): try: df.set_value(i, name, g['weather'][name]) except: pass if g.get('author'): for name in author_list: if g.get('author').get(name): try: df.set_value(i, name, g['author'][name]) except: pass for name in name_list: if g.get(name): try: if name is 'local_start_time': df.set_value(i, name, g[name][0:19]) else: df.set_value(i, name, g[name]) except: pass
I'd appreciate any and all suggestions for how to improve the speed here. Also I am using the
try...except because once in a while the encoding on some funky character is causing a