Splitting many data files by quintile

Question

The program below is about finding the values below the percentage and splitting them into five different output files. This works great for small input data files. But when I am trying to run it for a big data it takes ages to compute. Any suggestions on optimizing the program so that I can get fast computation?

#seperate by col 5
import numpy as np
import os
import re
import math
import csv
import sys
import time
import multiprocessing
from scipy import stats

curdir = 'Tools/'
inputdir = 'input/'
filename = 'input'
outputdir = 'output/'

def seperate(filename):
    print "Begin seperating File: %s" % filename
    dtype = np.dtype([('id1',str,12),('id2',str,12),('c1',int),('c2',int),('c3',float)])
    data = np.loadtxt(curdir + inputdir + filename,dtype=dtype)
    col = 'c3'
    col_data = np.array(data[:][col])
    cache = {}
    def get_per(per):
            if(cache.has_key(per)):
                return cache.get(per)
            else:
                r = 100 - stats.percentileofscore(col_data,per)
                cache[per] = r
                return r
    o1 = file(curdir + outputdir+ filename + '_0_20','w')
    o2 = file(curdir + outputdir+ filename + '_20_40','w')
    o3 = file(curdir + outputdir+ filename + '_40_60','w')
    o4 = file(curdir + outputdir+ filename + '_60_80','w')
    o5 = file(curdir + outputdir+ filename + '_80_100','w')
    for line in data:
        per = get_per(line[col])
        output_format = "%s %s %d %d %.1f\n"
        output_data = (line['id1'],line['id2'],line['c1'],line['c2'],line['c3'])
        if per>=0 and per < 20:
            o1.write(output_format % output_data)
        elif per>=20 and per<40:
            o2.write(output_format % output_data)
            pass
        elif per>=40 and per<60:
            o3.write(output_format % output_data)
            pass
        elif per>=60 and per<80:
            o4.write(output_format % output_data)
            pass
        elif per>=80 and per<=100:
            o5.write(output_format % output_data)
            pass
    o1.close()
    o2.close()
    o3.close()
    o4.close()
    o5.close()

    print "Finish seperating File: %s" % filename


ps = []
print "Reading Files"
for parent,dirNames,fileNames in os.walk(curdir+inputdir):
    if fileNames:
        #multi_process
        print fileNames
        for fileName in fileNames:
            m = re.match(r'^(?!\.)',fileName)
            if m:
                ps.append(multiprocessing.Process(target=seperate,args=(fileName,)))

#runing by multiple processes
for p in ps:
    p.start()
for p in ps:
    p.join()

Answer 1

• Calling percentileofscore for every distinct value is rather inefficient vs. the alternative of making just one call scoreatpercentile(col_data, [0,20,40,60,80,100]). You could directly compare the column values against the limits returned by that function.
• Apart from the above you are mostly doing disk I/O. You may have made matters worse by using multiprocessing. In particular, because you lauch as many processes as you have input files, you may have way too many processes competing for RAM and disk access.

Answer 2

I generated what I hope is sufficiently similar test data - five columns, where the last is a percentage and a float, and I ran your script through the line_profiler

kernprof -l -v ./script.py 

It seemed to spend most of its time in numpy's loadtxt, and the next highest amount of time in get_per, so I suspect these will be the two places to look at first.

Rewriting the code so it doesn't use loadtxt produces a noticeable increase (at least with the test data I generated, it would be good to see what test data you can use):

c3 = list()
with open( os.path.join(curdir,inputdir,filename)) as f:
    for line in f:
        c3.append(float(line.split()[4])) # Some assumption here about data delimiter
col_data = np.array(c3)

and then later, instead of for line in data:

with open(os.path.join(curdir,inputdir,filename)) as f:
    for line in f:

The output formatting will need adjustment, too.

Answer 3

It will not solve your problem of speeding it up, but instead of the next code

cache = {}
def get_per(per):
    if(cache.has_key(per)):
        return cache.get(per)
    else:
        r = 100 - stats.percentileofscore(col_data,per)
        cache[per] = r
        return r

I suggest a memoize decorator, which will make your code more readable, and will also make it so that you do not need to use this entire structure each time you want to memoize something:

import functools

def memoize(f):
    cache= {}
    @functools.wraps(f)
    def memf(*x):
        if x not in cache:
            cache[x] = f(*x)
        return cache[x]
    return memf

And then just use:

@memoize
def get_per(per):
    return 100 - stats.percentileofscore(col_data, per)

I know this does not answer your question of where the code can be sped up, but this is too long for a comment, and it significantly improves the readability of your code. The system of decorators (the @memoize) is a very practical one. For more info on decorators, i suggest https://stackoverflow.com/a/1594484/2393569