I have a dataframe (obtained from a csv saved from mysql) with several columns, and one of them consist of a string which is the representation of a json. The data looks like:
id email_id provider raw_data ts
1 aa@gmail.com A {'a':'A', 2019-23-08 00:00:00
'b':'B',
'c':'C'}
And what my desired output is:
email_id a b c
aa@gmail.com A B C
What I have coded so far is the following:
import pandas as pd
import ast
df = pd.read_csv('data.csv')
df1 = pd.DataFrame()
for i in range(len(df)):
dict_1 = ast.literal_eval(df['raw_content'][i])
df1 = df1.append(pd.Series(dict_1),ignore_index=True)
pd.concat([df['email_id'],df1])
This works but it has a very big problem: it is extremely low (it takes hours for 100k rows). How could I make this operation faster?
json.loads
be any faster? With a more limited structure it might. Another thing to consider is doing a string join on all thoseraw_data
strings, and callingloads
once. I haven't worked withjson
enough to know what's fast or slow in its parsing. \$\endgroup\$