# Splitting many data files by quintile

The program below is about finding the values below the percentage and splitting them into five different output files. This works great for small input data files. But when I am trying to run it for a big data it takes ages to compute. Any suggestions on optimizing the program so that I can get fast computation?

#seperate by col 5
import numpy as np
import os
import re
import math
import csv
import sys
import time
import multiprocessing
from scipy import stats

curdir = 'Tools/'
inputdir = 'input/'
filename = 'input'
outputdir = 'output/'

def seperate(filename):
print "Begin seperating File: %s" % filename
dtype = np.dtype([('id1',str,12),('id2',str,12),('c1',int),('c2',int),('c3',float)])
data = np.loadtxt(curdir + inputdir + filename,dtype=dtype)
col = 'c3'
col_data = np.array(data[:][col])
cache = {}
def get_per(per):
if(cache.has_key(per)):
return cache.get(per)
else:
r = 100 - stats.percentileofscore(col_data,per)
cache[per] = r
return r
o1 = file(curdir + outputdir+ filename + '_0_20','w')
o2 = file(curdir + outputdir+ filename + '_20_40','w')
o3 = file(curdir + outputdir+ filename + '_40_60','w')
o4 = file(curdir + outputdir+ filename + '_60_80','w')
o5 = file(curdir + outputdir+ filename + '_80_100','w')
for line in data:
per = get_per(line[col])
output_format = "%s %s %d %d %.1f\n"
output_data = (line['id1'],line['id2'],line['c1'],line['c2'],line['c3'])
if per>=0 and per < 20:
o1.write(output_format % output_data)
elif per>=20 and per<40:
o2.write(output_format % output_data)
pass
elif per>=40 and per<60:
o3.write(output_format % output_data)
pass
elif per>=60 and per<80:
o4.write(output_format % output_data)
pass
elif per>=80 and per<=100:
o5.write(output_format % output_data)
pass
o1.close()
o2.close()
o3.close()
o4.close()
o5.close()

print "Finish seperating File: %s" % filename

ps = []
for parent,dirNames,fileNames in os.walk(curdir+inputdir):
if fileNames:
#multi_process
print fileNames
for fileName in fileNames:
m = re.match(r'^(?!\.)',fileName)
if m:
ps.append(multiprocessing.Process(target=seperate,args=(fileName,)))

#runing by multiple processes
for p in ps:
p.start()
for p in ps:
p.join()

• You're writing line by line to 5 open files at the same time? Consider building an array of strings and then write the entire array at once, after you've read all the lines. Or is the array too big to hold in memory? Another option would be to sort the input file somehow and then output to one file at a time? – user1149 Oct 29 '14 at 17:25
• @BarryCarter Its too big to put in buffer.. So I had to chose line by line writing. – Sitz Blogz Oct 29 '14 at 17:29
• what happens if you run python -mcProfile your_script.py where script runs just one call seperate(filename) (no multiprocessing)? – jfs Oct 29 '14 at 17:30
• @J.F.Sebastian Past 20 mins its still processing. I dont see any difference. – Sitz Blogz Oct 29 '14 at 17:56
• @SitzBlogz: create an artifical data then. It is hard to optimize the code if you can't measure how fast it is. – jfs Oct 29 '14 at 18:28

• Calling percentileofscore for every distinct value is rather inefficient vs. the alternative of making just one call scoreatpercentile(col_data, [0,20,40,60,80,100]). You could directly compare the column values against the limits returned by that function.
• Apart from the above you are mostly doing disk I/O. You may have made matters worse by using multiprocessing. In particular, because you lauch as many processes as you have input files, you may have way too many processes competing for RAM and disk access.
• This sound more simpler. – Sitz Blogz Oct 31 '14 at 13:19

I generated what I hope is sufficiently similar test data - five columns, where the last is a percentage and a float, and I ran your script through the line_profiler

kernprof -l -v ./script.py


It seemed to spend most of its time in numpy's loadtxt, and the next highest amount of time in get_per, so I suspect these will be the two places to look at first.

Rewriting the code so it doesn't use loadtxt produces a noticeable increase (at least with the test data I generated, it would be good to see what test data you can use):

c3 = list()
with open( os.path.join(curdir,inputdir,filename)) as f:
for line in f:
c3.append(float(line.split()[4])) # Some assumption here about data delimiter
col_data = np.array(c3)


and then later, instead of for line in data:

with open(os.path.join(curdir,inputdir,filename)) as f:
for line in f:


The output formatting will need adjustment, too.

It will not solve your problem of speeding it up, but instead of the next code

cache = {}
def get_per(per):
if(cache.has_key(per)):
return cache.get(per)
else:
r = 100 - stats.percentileofscore(col_data,per)
cache[per] = r
return r


I suggest a memoize decorator, which will make your code more readable, and will also make it so that you do not need to use this entire structure each time you want to memoize something:

import functools

def memoize(f):
cache= {}
@functools.wraps(f)
def memf(*x):
if x not in cache:
cache[x] = f(*x)
return cache[x]
return memf


And then just use:

@memoize
def get_per(per):
return 100 - stats.percentileofscore(col_data, per)


I know this does not answer your question of where the code can be sped up, but this is too long for a comment, and it significantly improves the readability of your code. The system of decorators (the @memoize) is a very practical one. For more info on decorators, i suggest https://stackoverflow.com/a/1594484/2393569