2
\$\begingroup\$

Is there any way to make this code run faster. I have to compare values in a text file which is written in the format:

1 1.12345

2 0.12345

3 3.45678...

The values are in billions and the code takes weeks to run. As far as I can see, the calculation (subtraction in my case) is the heavy step.

a=open("output.txt","w")

for i in range(1,num_lines+1): #num_lines are the number of lines in the file
    read=linecache.getline('file.txt',i).rstrip('\n') #reads line by line from a text file
    splitted=read.split()   #splits the string "read"

    for k in range(i+1,num_lines+1):        
        read1=linecache.getline('just_nodes.txt',k).rstrip('\n')
        splitted1=read1.split()

        alpha_temp=abs(float(splitted[1])-float(splitted1[1])) #heavy step

        alpha=round(alpha_temp,3)

        if (alpha==1.000):
            a.write(str(splitted1[0]) +' '+ str(splitted[0]) + "\n")
a.close()
\$\endgroup\$
  • 1
    \$\begingroup\$ Could you also show what is linecache? Thanks. \$\endgroup\$ – alecxe Mar 24 '17 at 18:21
  • 1
    \$\begingroup\$ Could you add if you are using python-2.7 or python-3.x? In the former range will create an actual list, so if your file is billions of lines long, this list will be as well. \$\endgroup\$ – Graipher Mar 25 '17 at 11:27
  • 1
    \$\begingroup\$ I don't think that you have adequately explained what this code does. What do the data in the two files represent? What is linecache? What is the point of looking for values that differ by 1.000? \$\endgroup\$ – 200_success Mar 25 '17 at 14:38
3
\$\begingroup\$

One observation is that you are converting the splitted[1] to float inside the inner loop - redoing the same exact thing multiple times - do it once before the loop:

splitted = float(read.split(" ", 1)[1])  # convert to float here
for k in range(i + 1, num_lines + 1):        
    read1 = linecache.getline('just_nodes.txt', k).rstrip('\n')
    splitted1 = float(read1.split(" ", 1)[1])

    alpha_temp = abs(splitted - splitted1)

Note the other thing I've done - splitting by an explicit space and doing it strictly once - please check if it actually has a positive impact on performance.

You may also not strip the newlines from the end of the lines, float() can handle it:

In [1]: float("1.00\n")
Out[1]: 1.0
\$\endgroup\$
3
\$\begingroup\$

Going further on @alexce observation, you convert strings to floats on the second file for each line in the first file... That's way too much. Instead you should convert both files once and work only with floats after that.

You also should use open in a with statement to properly close your output file whatever happens.

I also don't know a thing about linecache but you may be able to use regular file iteration to simplify the problem at hand.

You may also want to use itertools.islice to simplify positionning yourself at position n into a list without having to take a slice (and thus copying data).

All in all, I'd write:

from itertools import islice


def parse_file(filename):
    with open(filename) as f:
        for line in f:
            id, alpha = line.split()
            yield id, float(alpha)


def ids_of_relevant_alphas(file1, file2):
    file2 = list(file2)
    for n, (id1, alpha1) in enumerate(file1, 1):
        # position ourselves at the nth line in the second file
        ifile2 = iter(file2)
        next(islice(ifile2, n, n), None)
        for id2, alpha2 in ifile2:
            alpha = round(abs(alpha1 - alpha2), 3)
            if alpha == 1.000:
                yield id1, id2


def main():
    file1 = parse_file('file.txt')
    file2 = parse_file('just_nodes.txt')
    with open('output.txt', 'w') as output:
        for id1, id2 in ids_of_relevant_alphas(file1, file2):
            output.write('{} {}\n'.format(id1, id2))


if __name__ == '__main__':
    main()
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.