5
\$\begingroup\$

I need to iterate through a file of words and perform tests on them.

import multiprocessing
import datetime

class line_Worker(object):
    def worker(self, chunks):
        for l in chunks:
            num = 0
            for num in range(100):
                print l, num
        return

if __name__ == '__main__':
    lines = [line.rstrip('\n') for line in open('words.txt')] #opens file and saves to list
    chunkNum = len(lines) / 26
    wordLists = [lines[x:x + chunkNum] for x in range(0, len(lines), chunkNum)]#divides list by 4 and saves to list of lists
    jobs = []
    timeStart = str(datetime.datetime.now())

    for i in range(27):
        p = multiprocessing.Process(target=line_Worker().worker, args=(wordLists[i],))
        jobs.append(p)
        p.start()
    for p in jobs:
        p.join()  # wait for the process to finish

    timeStop = str(datetime.datetime.now())

    print timeStart
    print timeStop

I have a problem set of 35,498,500 individual lines, and I need to try to get it to run in roughly 3 minutes. The current run times are 16 minutes.

Is there any way to speed this up? Thanks all!

\$\endgroup\$
  • 6
    \$\begingroup\$ If you are trying to be performant, why are you printing? \$\endgroup\$ – Stephen Rauch Mar 22 '17 at 1:42
  • 1
    \$\begingroup\$ Beside the obvious performance issue when you print, you're printing in a multi-threaded setting. That means that there is no way to determine the order in which stuff appears on screen. Is that fine? \$\endgroup\$ – ChatterOne Mar 22 '17 at 14:00
  • \$\begingroup\$ Have you measured which part of those 16 mins is spent in the reading in of the words.txt file? \$\endgroup\$ – Graipher Mar 23 '17 at 9:45
  • \$\begingroup\$ Its negligible sub 1 second, Im gonna try taking the print statements out and see what happens. I will post back. \$\endgroup\$ – Kyle Sponable Mar 23 '17 at 15:48
  • \$\begingroup\$ The purpose of the script is to demonstrate what a dictionary attack would look like to illustrate an internet safety demo \$\endgroup\$ – Kyle Sponable Mar 23 '17 at 15:50
1
\$\begingroup\$

Getting rid of the print statement fixed it thanks all!

import multiprocessing
import datetime

class line_Worker(object):
    def worker(self, chunks):
        for l in chunks:
            num = 0
            for num in range(100):
                print l, num#clearing this made the time 4 seconds thanks all!!
        return

if __name__ == '__main__':
    lines = [line.rstrip('\n') for line in open('words.txt')] #opens file and saves to list
    chunkNum = len(lines) / 26
    wordLists = [lines[x:x + chunkNum] for x in range(0, len(lines), chunkNum)]#divides list by 4 and saves to list of lists
    jobs = []
    timeStart = str(datetime.datetime.now())

    for i in range(27):
        p = multiprocessing.Process(target=line_Worker().worker, args=(wordLists[i],))
        jobs.append(p)
        p.start()
    for p in jobs:
        p.join()  # wait for the process to finish

    timeStop = str(datetime.datetime.now())

    print timeStart
    print timeStop
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.