Hashing with python optimization, hashlib and multiprocessing

Question

I am looking to get any type of improvement, I want to be able to hash a list of 'blocks' aka strings and check if hashes match per block, if you can help me with that, currently reading and writing is optimized as I am writing direct bytes, any type of improvement in performance is appreciated! I am also looking to use as little ram, when I read the file for confirmation I want to use as little ram as possible.

import multiprocessing
import time as t
from hashlib import sha256


def crypto(args: list):

    block, requests = args
    h = sha256(block).digest
    n = b'\n'
    with open("blocks.txt", "ab") as blocks:
        write = blocks.write
        for i in range(requests):
            write(h() + n)
        blocks.close()


if __name__ == "__main__":
    print("")
    print("Program starting... \n")
    last_block, block, counter, requests, threads, confirmed = [], "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()".encode(
        'UTF-8'), 0, 10000000, 3, False
    args = [block, requests]
    print("Clearing blocks.txt...")
    with open("blocks.txt", "wb") as blocks:
        blocks.write(b"")
        blocks.close()
    print("Cleared blocks.txt... \n")
    print("Starting the pool...")
    p = multiprocessing.Pool(processes=threads)
    print("Pool has started... \n")
    start_time = t.perf_counter()
    print("Starting the hashing...")
    p.map(crypto, [args] * threads)
    p.close()
    print("Hashed all requests... \n")
    time = t.perf_counter() - start_time
    print("Checking confirmation cycle...")
    with open("blocks.txt", "rb") as blocks:
        data = blocks.readlines()
        for sha in data:
            if data[0] == sha:
                confirmed = True
            else:
                print(f"**Invalid Confirmation:{sha}**")
                confirmed = False
                break
        blocks.close()
        confirm = t.perf_counter() - time
    print("Checked the confirmation cycle... \n")
    if confirmed:
        print("Valid confirmation cycle! \n")
    else:
        print("Invalid confirmation cycle! \n")
    print(
        f"Total time to process the {requests} requests with {threads} confirmations: {time}.")
    print(
        f"Average hashes per second per confirmation thread: {requests / time}."
    )
    print(f"Time to check if confirmations are legal: {confirm}.")

Answer 1

An alternative algorithm

I do not quite know what you want to do, and so this might miss. But I am presuming that you are not trying to stress-test sha256 by hashing the same block a zillion times and checking it works, and thus your h(), which always returns the same thing, would be better modelled as something like this:

from random import randint

def randomfail_hasher(block):
    if randint(0, 100_000_000) < 2:
        return sha256(block + b"Some other data").digest()
    else:
        return sha256(block).digest()

Note that you can write very long numbers like this: 100_000_000_000 which is extremely useful if you are not very good at counting large numbers of zeros, like me. (Before I knew that I used to do ugly things like int(1E9).)

Per your comments on the SO question, you want the whole thing to bail the moment a match doesn't happen. Although you apparently want to keep the hashes, I can't see any reason to do that as they are either the same as the correct target or not, so we are just duplicating the same values.

Here is my crypto function:

class ValidationError(Exception):
    pass

def crypto(args):
    block, requests, correct = args
    for candidate in (randomfail_hasher(block) for _ in range(requests)):
        if candidate != correct:
            raise ValidationError(candidate)

This does the hashing and validation in one step, for a given target. It throws if validation ever fails.

As I understand it, the entire pool has one target (certainly that's currently true). So we want to kill the whole pool if anything fails. For this I use an explicitly named callback function (a lambda would be fine, but this is clearer and you may want to do something else:

def die(_):
    p.terminate()

And then I use map_async:

if __name__ == "__main__":
    block = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()".encode(
        "UTF-8"
    )
    requests = 10_000_000
    threads = multiprocessing.cpu_count()

    p = multiprocessing.Pool(processes=threads)

    start_time = t.perf_counter()
    print("Starting the hashing...")

    correct = sha256(block).digest()
    try:
        res = p.map_async(
            crypto, [(block, requests, correct)] * threads, error_callback=die
        )
        res.get()
        p.close()
        print("Success")
    except ValidationError as e:
        print(f"Failed: {e}")

    time = t.perf_counter() - start_time

    print(
        f"Total time to process the {requests} requests with {threads} confirmations: {time}."
    )

I have changed a few other things here:

• we have as many threads as cpus, which is probably right for a hashing problem
• I assign variables directly, rather than unpacking a tuple (!)
• I use immutables (tuples) instead of lists.

Timings

On my machine, your code took (with 6 rather than 3 threads for the cpus here):

Total time to process the 10000000 requests with 6 confirmations: 11.103364465001505.
Average hashes per second per confirmation thread: 900627.916116833.
Time to check if confirmations are legal: 92861.462768207.

Replacing lists with tuples, removing redundant close() got

Total time to process the 10000000 requests with 6 confirmations: 10.663555920997169.
Average hashes per second per confirmation thread: 937773.4851382372.
Time to check if confirmations are legal: 93723.683743126.

That's probably significant, but I didn't repeat it.

The alternative algorithm here takes ~34 seconds to validate everything, and is of course much cheaper in the event of a failure.

If you mean to test different targets for each thread that is perfectly possible too, and will still be faster than writing and reading. You can write the target hashes to disk if you really need to.