Proper parallelism is hard in any language. You should definitely take the time to at least skim through all of the python docs for both the
threading module AND the
multiprocessing module. (see details below)
Also, in python there's an added wrinkle in that there is a Global Interpreter Lock. The simplified explanation is that Threads from the threading module run asynchronously, but not actually in parallel while multiprocessing Processes run in true parallel. For code like yours where you're querying the web and waiting for IO, threading helps because you can get something else done while waiting. But you might also get a speedup from dividing the load with multiprocessing (and have several threads inside each of those). Mulitprocessing processes take longer to setup so they're not as good when you're waiting on IO (as you are with
response.get() ), but they run in true parallel--different processes in the system.
I'm not an expert on tuning between the two yet. I'll just warn you that when there are identically named communication objects like Queue and Event inside both the threading and multiprocessing module, and you have to use the correct one for the boundary your communication is crossing or it won't work.
Usually you want try statements to encapsulate the minimum amount of code possible so you can better predict what to do with the errors. As such
response = response.get(... is probably the only line currently inside that actually needs to be in the try statement.
Edit: this section originally used dict keys which are only good for exact matches.
Removing specific responses can be done in one loop instead of going back over the loop many times. Also, the column of
if statements can be made into a single statement by looking at a list. Something like:
removelist = ["You are using", "Google Chrome", "Tweet", "wa-com.com"]
for site in sites:
for x in removelist:
if x in site:
Writing to a text file via a command line passed into os.system is not the most efficient method of doing that. First because you're involving the os module as a middle man when you don't need to, and second because you're doing all of this from up to 40M+ threads at once.
Instead write the output from each thread to a python Queue.
for site in sites:
Then have a single thread (or your main thread) read from the Queue and write to the file. Something like
with open('output.txt') as f:
while not all_done_event.is_set():
Next you probably don't want 40M+ threads going at once, but
time.sleep(0.5) is not the best way to control the number of threads launched. Instead determine the approximate maximum number of threads you can launch before the first one gets a response, and use a Semaphore object to control when new threads are launched. This keeps you always at the optimal number until you are done rather than using a guess about how long each thread will run. Something like:
maxthreads = 15
threadpool = BoundedSemaphore(value=maxthreads)
for ip in ips:
scan = ip2host(ip)
threadpool.release() at the end of the
ip2host.run() function to signal that the next thread can be started. Again, you may also want to split
ips groups and wrap the threading objects in a multiprocessing Pool class object to get true parallelism on top of threading.
Hope this helps!