4
\$\begingroup\$

I have the following program. I have 90 million domain names split between 87 files. I need to resolve the IP of all. Right now I feel like this is super slow.

import multiprocessing
import fileinput
import socket
from datetime import datetime

socket.setdefaulttimeout(2)

def worker(i):
    filename = (str(i).zfill(len(str(111))))
    buffer_array = []
    for line in fileinput.input(f"./data/{filename}"):
        domain = '.'.join( list( reversed( line.split('\t')[1].split('.') ) ) )
        try:
            ip_list = socket.gethostbyname(domain)
            buffer_array.append(ip_list)
            if(len(buffer_array) == 1000):
                with open(f"./parsed_data/{i}", "a+") as save_file:
                    save_file.write("\n".join(buffer_array))
                    buffer_array = []
                    print(f"{datetime.now()} -- WRITING: worker {i}")
                    save_file.close()

        except Exception as e:
            pass

    with open(f"./parsed_data/{i}", "a+") as save_file:
        save_file.write("\n".join(buffer_array))
        print(f"{datetime.now()} -- WRITING ***FINAL***: worker {i}")
        save_file.close()

if __name__ == '__main__':
    jobs = []
    for i in range(87):
        p = multiprocessing.Process(target=worker, args=(i,))
        jobs.append(p)
        p.start()

In node, I was able to do 1,000,000 on a 2017 macbook pro in about 12 hours without using workers. Right now this accomplished 39 million in 32 hours. My hope was I could make it do it's thing in 12 hours (1m per worker in in 12 hours) on a AWS T3aXL

Does anyone have a bit faster way of doing this?

\$\endgroup\$
7
  • 4
    \$\begingroup\$ Surely the bottleneck is speed of DNS lookups. \$\endgroup\$ Oct 5, 2020 at 1:29
  • 2
    \$\begingroup\$ where is multiprocessing being used? \$\endgroup\$
    – hjpotter92
    Oct 5, 2020 at 8:23
  • \$\begingroup\$ @hjpotter92 I forgot it lol. \$\endgroup\$
    – Quesofat
    Oct 5, 2020 at 13:55
  • \$\begingroup\$ I'm wondering if there's a way to do this faster with an async library. \$\endgroup\$
    – Quesofat
    Oct 5, 2020 at 13:56
  • \$\begingroup\$ Generally, use threads or async for I/O bound code such as yours. See ThreadPoolExecutor or maybe asyncio.gather. \$\endgroup\$
    – RootTwo
    Oct 5, 2020 at 18:27

1 Answer 1

3
\$\begingroup\$

Minor Python cleanup

First, a bunch of little stuff:

This -

(str(i).zfill(len(str(111))))

does not need outer parens. Also, why the acrobatics to get "3"? Just declare a MAX_DIGITS = 3.

This -

'.'.join( list( reversed( line.split('\t')[1].split('.') ) ) )

should not be creating an inner list. The output of reversed is iterable.

This -

if(len(buffer_array) == 1000):

does not need outer parens; you're not in (C/Java/etc).

This -

save_file.close()

needs to be deleted both places that it appears. There's an implicit close() from your use of a with.

This -

    except Exception as e:
        pass

is a bad idea. Log or output the exception.

87 deserves a named constant.

Broad performance

gethostbyname is probably slow (as indicated in the comments). Consider running a local caching DNS resolution service to eliminate the first network hop.

\$\endgroup\$
3
  • \$\begingroup\$ I appreciate it. I'm not a Python dev. I'm a node / ruby dev. What's this about local caching DNS resolution? Do you have a link to some literature? \$\endgroup\$
    – Quesofat
    Oct 11, 2020 at 15:56
  • \$\begingroup\$ nlnetlabs.nl/projects/unbound/about \$\endgroup\$
    – Reinderien
    Oct 11, 2020 at 17:51
  • \$\begingroup\$ Thanks mate! Cheers \$\endgroup\$
    – Quesofat
    Oct 12, 2020 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.