Purpose of the question: learn more about ways to implement concurrency in Python / experimenting.
Context: I want to count all of the words in all of the files that match a particular pattern. The idea is that I can invoke the function count_words('/foo/bar/*.txt')
and all of the words (i.e., strings separated by one or more whitespace characters) will be counted.
In the implementation I am looking for ways to implement count_words
using concurrency. So far I managed to use multiprocessing
and asyncio
.
What do you think of these two approaches? Do you see alternative approaches to do the same task? What about the code style, should I implement a dedicated class for each approach?
I did not use threading
as I noticed the performance improvement was not that impressive due to the limitation of the Python GIL.
import asyncio
import multiprocessing
import time
from pathlib import Path
from pprint import pprint
def count_words(file):
with open(file) as f:
return sum(len(line.split()) for line in f)
async def count_words_for_file(file):
with open(file) as f:
return sum(len(line.split()) for line in f)
def async_count_words(path, glob_pattern):
event_loop = asyncio.get_event_loop()
try:
print("Entering event loop")
for file in list(path.glob(glob_pattern)):
result = event_loop.run_until_complete(count_words_for_file(file))
print(result)
finally:
event_loop.close()
def multiprocess_count_words(path, glob_pattern):
with multiprocessing.Pool(processes=8) as pool:
results = pool.map(count_words, list(path.glob(glob_pattern)))
pprint(results)
def sequential_count_words(path, glob_pattern):
for file in list(path.glob(glob_pattern)):
print(count_words(file))
if __name__ == '__main__':
benchmark = []
path = Path("../data/gutenberg/")
# no need for benchmark on sequential_count_words, it is very slow!
# sequential_count_words(path, "*.txt")
start = time.time()
async_count_words(path, "*.txt")
benchmark.append(("async version", time.time() - start))
start = time.time()
multiprocess_count_words(path, "*.txt")
benchmark.append(("multiprocess version", time.time() - start))
print(*benchmark)
For simulating large quantity of files, I downloaded some books from Project Gutenberg (https://gutenberg.org/) and used the following command to create several duplicates of the same file.
for i in {000..99}; do cp 56943-0.txt $(openssl rand -base64 12)-$i.txt; done