Simple concurrency implemented in Python

Question

Purpose of the question: learn more about ways to implement concurrency in Python / experimenting.

Context: I want to count all of the words in all of the files that match a particular pattern. The idea is that I can invoke the function count_words('/foo/bar/*.txt') and all of the words (i.e., strings separated by one or more whitespace characters) will be counted.

In the implementation I am looking for ways to implement count_words using concurrency. So far I managed to use multiprocessing and asyncio.

What do you think of these two approaches? Do you see alternative approaches to do the same task? What about the code style, should I implement a dedicated class for each approach?

I did not use threading as I noticed the performance improvement was not that impressive due to the limitation of the Python GIL.

import asyncio
import multiprocessing
import time
from pathlib import Path
from pprint import pprint


def count_words(file):
    with open(file) as f:
        return sum(len(line.split()) for line in f)


async def count_words_for_file(file):
    with open(file) as f:
        return sum(len(line.split()) for line in f)


def async_count_words(path, glob_pattern):
    event_loop = asyncio.get_event_loop()
    try:
        print("Entering event loop")
        for file in list(path.glob(glob_pattern)):
            result = event_loop.run_until_complete(count_words_for_file(file))
            print(result)
    finally:
        event_loop.close()


def multiprocess_count_words(path, glob_pattern):
    with multiprocessing.Pool(processes=8) as pool:
        results = pool.map(count_words, list(path.glob(glob_pattern)))
        pprint(results)


def sequential_count_words(path, glob_pattern):
    for file in list(path.glob(glob_pattern)):
        print(count_words(file))


if __name__ == '__main__':
    benchmark = []
    path = Path("../data/gutenberg/")
    # no need for benchmark on sequential_count_words, it is very slow!
    # sequential_count_words(path, "*.txt")

    start = time.time()
    async_count_words(path, "*.txt")
    benchmark.append(("async version", time.time() - start))

    start = time.time()
    multiprocess_count_words(path, "*.txt")
    benchmark.append(("multiprocess version", time.time() - start))

    print(*benchmark)

For simulating large quantity of files, I downloaded some books from Project Gutenberg (https://gutenberg.org/) and used the following command to create several duplicates of the same file.

for i in {000..99}; do cp 56943-0.txt $(openssl rand -base64 12)-$i.txt; done

Answer 1

async def count_words_for_file(file):
    with open(file) as f:
        return sum(len(line.split()) for line in f)

Just making this function async won't bring you any benefit. asyncio can improve performance only if you're running multiple operations that can be parallelized. Usual case is when you're running multiple downloads: asyncio will do some job instead of idle waiting answer from network and through that bring you benefit.

In your case almost everything faces CPU (splitting lines) and disk I/O (reading files). For can be parallelized throught processes only (due to GIL), for second you can use threads (since GIL doesn't affect disk I/O). In both cases you can use asyncio to rule the process with run_in_executor.

Read this link for general explanation and few examples.