# Speeding up I/O bound python program

I have the following program. I have 90 million domain names split between 87 files. I need to resolve the IP of all. Right now I feel like this is super slow.

import multiprocessing
import fileinput
import socket
from datetime import datetime

socket.setdefaulttimeout(2)

def worker(i):
filename = (str(i).zfill(len(str(111))))
buffer_array = []
for line in fileinput.input(f"./data/{filename}"):
domain = '.'.join( list( reversed( line.split('\t')[1].split('.') ) ) )
try:
ip_list = socket.gethostbyname(domain)
buffer_array.append(ip_list)
if(len(buffer_array) == 1000):
with open(f"./parsed_data/{i}", "a+") as save_file:
save_file.write("\n".join(buffer_array))
buffer_array = []
print(f"{datetime.now()} -- WRITING: worker {i}")
save_file.close()

except Exception as e:
pass

with open(f"./parsed_data/{i}", "a+") as save_file:
save_file.write("\n".join(buffer_array))
print(f"{datetime.now()} -- WRITING ***FINAL***: worker {i}")
save_file.close()

if __name__ == '__main__':
jobs = []
for i in range(87):
p = multiprocessing.Process(target=worker, args=(i,))
jobs.append(p)
p.start()


In node, I was able to do 1,000,000 on a 2017 macbook pro in about 12 hours without using workers. Right now this accomplished 39 million in 32 hours. My hope was I could make it do it's thing in 12 hours (1m per worker in in 12 hours) on a AWS T3aXL

Does anyone have a bit faster way of doing this?

• Surely the bottleneck is speed of DNS lookups. Oct 5, 2020 at 1:29
• where is multiprocessing being used? Oct 5, 2020 at 8:23
• @hjpotter92 I forgot it lol. Oct 5, 2020 at 13:55
• I'm wondering if there's a way to do this faster with an async library. Oct 5, 2020 at 13:56
• Generally, use threads or async for I/O bound code such as yours. See ThreadPoolExecutor or maybe asyncio.gather. Oct 5, 2020 at 18:27

## Minor Python cleanup

First, a bunch of little stuff:

This -

(str(i).zfill(len(str(111))))


does not need outer parens. Also, why the acrobatics to get "3"? Just declare a MAX_DIGITS = 3.

This -

'.'.join( list( reversed( line.split('\t')[1].split('.') ) ) )


should not be creating an inner list. The output of reversed is iterable.

This -

if(len(buffer_array) == 1000):


does not need outer parens; you're not in (C/Java/etc).

This -

save_file.close()


needs to be deleted both places that it appears. There's an implicit close() from your use of a with.

This -

    except Exception as e:
pass


is a bad idea. Log or output the exception.

87 deserves a named constant.

gethostbyname is probably slow (as indicated in the comments). Consider running a local caching DNS resolution service to eliminate the first network hop.