# Simple concurrency implemented in Python

Context: I want to count all of the words in all of the files that match a particular pattern. The idea is that I can invoke the function count_words('/foo/bar/*.txt') and all of the words (i.e., strings separated by one or more whitespace characters) will be counted.

In the implementation I am looking for ways to implement count_words using concurrency. So far I managed to use multiprocessing and asyncio.

What do you think of these two approaches? Do you see alternative approaches to do the same task? What about the code style, should I implement a dedicated class for each approach?

I did not use threading as I noticed the performance improvement was not that impressive due to the limitation of the Python GIL.

import asyncio
import multiprocessing
import time
from pathlib import Path
from pprint import pprint

def count_words(file):
with open(file) as f:
return sum(len(line.split()) for line in f)

async def count_words_for_file(file):
with open(file) as f:
return sum(len(line.split()) for line in f)

def async_count_words(path, glob_pattern):
event_loop = asyncio.get_event_loop()
try:
print("Entering event loop")
for file in list(path.glob(glob_pattern)):
result = event_loop.run_until_complete(count_words_for_file(file))
print(result)
finally:
event_loop.close()

def multiprocess_count_words(path, glob_pattern):
with multiprocessing.Pool(processes=8) as pool:
results = pool.map(count_words, list(path.glob(glob_pattern)))
pprint(results)

def sequential_count_words(path, glob_pattern):
for file in list(path.glob(glob_pattern)):
print(count_words(file))

if __name__ == '__main__':
benchmark = []
path = Path("../data/gutenberg/")
# no need for benchmark on sequential_count_words, it is very slow!
# sequential_count_words(path, "*.txt")

start = time.time()
async_count_words(path, "*.txt")
benchmark.append(("async version", time.time() - start))

start = time.time()
multiprocess_count_words(path, "*.txt")
benchmark.append(("multiprocess version", time.time() - start))

print(*benchmark)


For simulating large quantity of files, I downloaded some books from Project Gutenberg (https://gutenberg.org/) and used the following command to create several duplicates of the same file.

for i in {000..99}; do cp 56943-0.txt $(openssl rand -base64 12)-$i.txt; done


async def count_words_for_file(file):

Just making this function async won't bring you any benefit. asyncio can improve performance only if you're running multiple operations that can be parallelized. Usual case is when you're running multiple downloads: asyncio will do some job instead of idle waiting answer from network and through that bring you benefit.