# Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have others.. but all rows put together have 1142 individual parameters.) and so I've written the following script, which works, but the sys, AWS EC2 instance, shut it down after about 64% of its execution.. The output just stopped and terminal shows "Killed". The systems guys at my company told me its a memory issue and it was killed by the os.
Can someone help me make this more memory efficient?

The raw data for this process is roughly 1GB in size.. I load it from a csv into a pandas dataframe of 2.2+ Mn rows and 11 columns.. 1 column has links- prbably around 60 - 80 chars, most of them. Two other columns have either a dict with key/value pairs or a get url with its set of parameters. The other columns have only null or simple short texts. I'm trying to get these parameters from the GET urls and links into their own individual dataframe columns- for most parameters I suspect most rows will have nulls but nonetheless I want this in seperate clean row column structure so I can proceed to analyze this data.

My server has 32Gb of memory.

The code

import pandas as pd, numpy as np, datetime, ast
import sys

print qdata.shape

fl = open('q_unique_params.txt','r')
cols = ast.literal_eval(cols)
print type(cols)
print cols
fl.close()

global key_
def get_params(rw):
try:
return rw['params'][key_]
except KeyError:
print "KeyError"
return ""
except Exception as e:
print key_, e
return ""

f = open('q_params_parsed.txt','w')
f.write('\n')
f.close()

for col in cols:
key_=col
print key_
qdata["{}_param".format(key_)]=qdata.apply(get_params)
f = open('qtr_params_parsed.txt','a')
f.write('\n' + str(key_))
f.close()
print qdata.shape

print qdata.shape
qdata.to_csv('q_data_parsed_to_col_full.csv')

• You suspect that your program fails due to a memory issue. Are you able to reduce your input CSV from millions of rows down to a hundred or so and run the program to completion? – Mathias Ettinger Jun 27 '18 at 9:04
• Yes.. I test in limited format, before the run. But i tested to ~1k rows, so looping through 1k rows at a time, will take ages again, which is why I ran it altogether. Perhaps I can determine a good amount of rows to loop through a time, but thats an unnecessary problem if you ask me. I was wondering if this problem could be improved to be made memory efficient in itself. – James Kumar Jun 27 '18 at 9:29
• I was just making sure that the question is on-topic as code that does not run to completion is off-topic for Code Review. If the code run fine on small inputs, you indeed have a scalability issue, but we can accept the question. – Mathias Ettinger Jun 27 '18 at 9:32