Checking hash and passwords with a wordlist, more efficient

Question

I have done a small code in which with a wordlist (out68.lst) I get the passwords from the hashes in the file 'shadow3'.

import crypt
import string
import itertools
import datetime

dir = "shadow3"                          #File that contains hashes and users
file = open(dir, 'r').readlines()

username = []
hashed = []
k=0


for x in file:
    usr, hshd, wtf, iss, this, thing, here, doing, example = x.split(':')
    username.append(usr)
    hashed.append(hshd)
#Loop in order to split the data in the file and store it in username and hashed

grupo1=open('out68.lst','r').readlines()
long=len(grupo1)
print(long)
for y in grupo1:    #Loop in order to go through all the possible words available
  c = 0
  y=y.rstrip('\n')
  y=y.capitalize()
  k = k+1
  if k==(long//100):
    print('1%')
  if k==(long//10):
    print('10%')
  if k==(long//5):
    print('20%')
  if k==(3*long//10):
    print('30%')
  if k==(4*long//10):     #Just to check the progress
    print('40%')
  if k==(5*long//10):
    print('50%')
  if k==(6*long//10):
    print('60%')
  if k==(7*long//10):
    print('70%')
  if k==(8*long//10):
    print('80%')
  if k==(9*long//10):
    print('90%')


  for x in hashed:
      rehashed = crypt.crypt(y, x) #Hash verification f(passwor+hash)=hash?
      if rehashed == x:
          print('La contraseña del usuario ' + username[c] + ' es ' + y)
      c = c + 1

It does work but depending on the size of the files, it can last now from 30 minutes to 6 hours. So I am asking if there is any way to improve the performance, by paralelization, or GPU processing (but I have no idea about this).

Answer 1

Some suggestions:

• Run the code through at least one linter such as flake8 or pycodestyle to produce more idiomatic code.
• Don't read all the lines into a variable before starting processing - this will slow things down and use much more memory than necessary for large files. Instead you can use for line in file.readlines().
• You are doing ten calculations in order to run a single print statement. Either get rid of them or do something simpler like print("{}/{} complete".format(k, long)).
• If you know y has exactly one newline at the end you can do y[:-1] instead of y.rstrip('\n').
• Capitalizing each word is expensive. Avoid it if at all possible.
• If you don't need a bunch of the fields in an input file add a limit to your split() and mark the last stuff as discarded by using the _ variable. For example: usr, hshd, _ = x.split(':', 3)
• Rather than keeping track of k manually you can just do for k, y in enumerate(grupo1).
• Rather than having a list of usernames and a list of their hashed passwords, a Dict[str, str] of username to hash should be easier to keep track of.

Answer 2

You should make sure you close files you open. This can be easily achieved using the with keyword:

file_name = "shadow3"
with open(file_name) as file:
    usernames, hashed_passwords = [], []
    for line in file:
        username, hashed_password, *_ = line.split(':')
        usernames.append(username)
        hashed_passwords.append(hashed_password)

Calling a file dir is just setting yourself up for trouble later. I also used the advanced tuple assignment by using * to assign the rest of the line to the unused variable _ (a customary name for unused variables).

Note that open opens a file in read-only mode by default, so 'r' is implicitly used.

---

Whenever you want to iterate over an iterable but also need a counter, use enumerate and whenever you want to iterate over two iterables in tandem, use zip:

with open('out68.lst') as group:
    length = len(group)
    for k, password in enumerate(group):
        password = password.rstrip('\n').capitalize()

        if k == length // 100 or k % (length // 10) == 0:
            print(f"{k / length:.%}")
        for username, hashed_password in zip(usernames, hashed_passwords):
            if crypt.crypt(password, hashed_password) == hashed_password:
                print(f'La contraseña del usuario {username} es {password}')

Here I also used modular arithmetic to cut down your special cases for ten percent increments, used the new f-string formatting.

---

You might want to add some functionality where you save username password pairs if you have found some and remove them from the usernames and hashed_passwords lists so you don't keep on checking them once you found the password. To do this efficiently it might be necessary to change your data format to {username: hashed_password} and {username: password} for the yet to find and already found usernames.

file_name = "shadow3"
users = {}
with open(file_name) as file:
    for line in file:
        username, hashed_password, *_ = line.split(':')
        users[username] = hashed_password

cracked_users = {}
with open('out68.lst') as group:
    length = len(group)
    for k, password in enumerate(group):
        password = password.rstrip('\n').capitalize()
        if k == length // 100 or k % (length // 10) == 0:
            print(f"{k / length:.%}")
        if not users:
             print("Cracked all passwords")
             break
        for username, hashed_password in users.items():
            if crypt.crypt(password, hashed_password) == hashed_password:
                print(f'La contraseña del usuario {username} es {password}')
                cracked_users[username] = password
                del users[username]

---

In general, don't be afraid of giving your variables clear names. If it is a username, call it username, not usr. If it is a hashed password, call it hashed_password, not hshd.