I need to iterate through a file of words and perform tests on them.
import multiprocessing import datetime class line_Worker(object): def worker(self, chunks): for l in chunks: num = 0 for num in range(100): print l, num return if __name__ == '__main__': lines = [line.rstrip('\n') for line in open('words.txt')] #opens file and saves to list chunkNum = len(lines) / 26 wordLists = [lines[x:x + chunkNum] for x in range(0, len(lines), chunkNum)]#divides list by 4 and saves to list of lists jobs =  timeStart = str(datetime.datetime.now()) for i in range(27): p = multiprocessing.Process(target=line_Worker().worker, args=(wordLists[i],)) jobs.append(p) p.start() for p in jobs: p.join() # wait for the process to finish timeStop = str(datetime.datetime.now()) print timeStart print timeStop
I have a problem set of 35,498,500 individual lines, and I need to try to get it to run in roughly 3 minutes. The current run times are 16 minutes.
Is there any way to speed this up? Thanks all!