3
\$\begingroup\$

I have two txt files. Both contain 60k+ words in a list. I put together some code to loop through each file and concatenate the strings

import hashlib
with open('1.txt') as f:
    a= [line.rstrip() for line in f]
with open('2.txt') as f:
    b= [line.rstrip() for line in f]

for y in a:
    for z in b:
        if hashlib.md5(y+z).hexdigest()=='XXXTHEXXXHASHXXX':
            print y+z
            break

So line 1 in file 1 gets concatenated with line 1 in file 2, then line 2 etc. Line 2 in file gets.... you get where this is going...

Is there a cleaner way to do this? Not only that but how can I edit the script to use more cores?

\$\endgroup\$
  • \$\begingroup\$ I'm curious — why are you doing this hash comparison? \$\endgroup\$ – 200_success Feb 4 '16 at 9:44
  • \$\begingroup\$ 0xf.at/play/20 \$\endgroup\$ – pee2pee Feb 4 '16 at 9:50
2
\$\begingroup\$

In general, no, this would not be the fastest approach. However, assuming that all the lines are quite small, I think you won't do better by the 'general' fast approach.

You could use hashlib.md5().copy().

import hashlib
with open('1.txt') as f:
    a= [line.rstrip() for line in f]
with open('2.txt') as f:
    b= [line.rstrip() for line in f]

for y in a:
    prefix_hash = hashlib.md5(y)
    for z in b:
        if prefix_hash.copy().update(z).hexdigest() == 'XXXTHEXXXHASHXXX':
            print y + z
            break

Again, assuming len(y) is not that large, this is unlikely to do any better, and might even do worse because of the .copy() now used. But you're welcome to benchmark it.

|improve this answer|||||
\$\endgroup\$
2
\$\begingroup\$

There's definitely improvement to be made here. Only one of these files needs to be saved in memory (the one you need to loop over multiple times) so you should just use

with open('2.txt') as f:
    b = [line.rstrip() for line in f]

And then iterate directly over the other file:

with open('1.txt') as f:
    for y in f:
        y = y.rstrip()
        for z in b:
            if hashlib.md5(y + z).hexdigest()=='XXXTHEXXXHASHXXX':
                print y + z
                break

Keeping 2 large files in memory can slow down your script badly, and rarely will you actually need to. In this case keeping one in memory does save time, but there's no need for both.


I have another suggestion that may help here but I'm less sure. You have an unusual situation where of all the possible combinations, only 1 is actually what you need (if I've understood correctly). On that basis, you could use the any operator to check whether or not you need to fully run your inner loop. any still runs a full loop, but it does it faster than plain Python does because the definition for any is written in C. You could run an any test to discover if any of the pairs with the current y work, and if they do then actually test each individual pair.

This is what I mean:

with open('1.txt') as f:
    for y in f:
        y = y.rstrip()

        if not any(hashlib.md5(y + z).hexdigest()=='XXXTHEXXXHASHXXX'
                   for z in b):
            continue

This any call will evaluate as True if there's a valid hash value for this value of y but False otherwise. This means that if y cant create the hash value then you will continue, which just means going on to the next iteration of the loop. However if y does create the valid hash at some point, then you should next run the original loop:

        if not any(hashlib.md5(y + z).hexdigest()=='XXXTHEXXXHASHXXX'
                   for z in b):
            continue

        for z in b:
            if hashlib.md5(y + z).hexdigest()=='XXXTHEXXXHASHXXX':
                print y + z
                break

Technically this means that on the correct value of y, you'll run the loop twice. Once to confirm you have the correct y and then another to actually find the correct z. But on all your invalid values (of which there will be a lot) then you will be running the test faster, so I believe this may improve your speed overall.

|improve this answer|||||
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.