I'm currently trying to process a corpus of a million patent text files, which contain about 10k non-unique words on average. My current data pipeline works as follows:

  1. Load the patent texts as strings from a local sqlite database table
  2. Tokenize¹ each document and save the results in a new table
  3. Create a dictionary containing all unique words in the corpus
  4. Train an tfidf model with the tokenized documents

¹Tokenization means taking a document text (string) as input and returning a list containing every word in the document (duplicates allowed). Words tend to be separated by spaces, special characters, numbers etc., the regex in my code has served quite well for this purpose.

In my data pipeline, I identified the tokenize function as my bottleneck, the relevant part is provided in my MWE below:

import re
import urllib.request
import time


PAT_ALPHABETIC = re.compile(r'[^\W\d]+')

def tokenize(text):
    for match in matches:
        yield match.group()

def preprocessing(doc):
    tokens = [token for token in tokenize(doc)]
    return tokens

start_time = time.time()
print("--- %s seconds ---" % (time.time() - start_time))
  • \$\begingroup\$ Is that a vocabulary of 10k distinct words per document (not mean, esp. with stemming), or a mere 10 k words per text file - say, about 100 k characters? I would not consider the latter very big, but a large amount again may amount to a considerable total. \$\endgroup\$
    – greybeard
    Apr 8, 2020 at 22:50
  • \$\begingroup\$ It's the latter one. I'm processing roughly a million patent texts which tend to have 10-20k (non-distinct) words per text on average. Due to the technical nature of patents, the number of distinct words also gets very large when considering the whole corpus. \$\endgroup\$ Apr 9, 2020 at 8:59
  • \$\begingroup\$ I'd expect insight into the overall problem more promising with more code context provided. Can you present, in your question (where information like patent texts would be placed advantageously), method and measurement results that let you [identify] the tokenize function as [the] bottleneck? \$\endgroup\$
    – greybeard
    Apr 9, 2020 at 10:12
  • \$\begingroup\$ Hip-shot: time WHITE_DIGITS = re.compile(r'[\W\d]+') WHITE_DIGITS.split(doc) \$\endgroup\$
    – greybeard
    Apr 9, 2020 at 10:24
  • 2
    \$\begingroup\$ Welcome to Code Review. Unfortunately, you may have misunderstood this site. A Minimum Working Example is something Stack Overflow loves (also known as MCVE). On Code Review however, we need to see the real deal. The actual code. So your question risks getting closed even though it's answered. Please keep this in mind for your next question (and no, please don't touch the code now an answer has come in). For future reference, please see our FAQ on How to get the best value out of Code Review - Asking Questions. \$\endgroup\$
    – Mast
    Apr 9, 2020 at 11:27

2 Answers 2


Some small performance can be gained by directly yielding from the iterator instead of having a for loop do it, using the yield from keywords, and using list() instead of a list comprehension:

def tokenize2(text):
    yield from PAT_ALPHABETIC.finditer(text)

def preprocessing2(doc):
    return list(tokenize2(doc))

For the given example document this gives about a 15% speed-up:

In [15]: %timeit preprocessing(doc)
335 ms ± 2.29 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [16]: %timeit preprocessing2(doc)
287 ms ± 2.79 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Slightly faster yet is not even having the preprocessing function and directly returning all tokens (this avoids one function call and lets the re do its best):

def tokenize3(text):
    return PAT_ALPHABETIC.findall(text)

This is about 35% faster then the code in the OP:

In [21]: %timeit tokenize3(doc)
217 ms ± 1.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Other than that, we can't really help you without seeing more of your script. You can probably parallelize this task by scanning multiple documents in parallel, and especially by making downloading and scanning asynchronous, so that you scan a document whenever it finishes downloading, but already download the next document(s) in the background.


I would guess that you're never going to take another 50% off. You could consider pulling the problem down into C or Haskell or Awk or whatever and then binding that more tightly optimized implementation back up into python; IDK.

As for more incremental improvements, it depends a bit how the thing is going to be used.

You flagged the question for multi-threading. That certainly might help, but you'll have to figure out how to divy up the job. Maybe there are natural split-off points that can be identified without actually reading the subject data, in which case any parallelization system will probably work. Maybe you can figure out the size of the subject on disk, and open multiple readers against that file starting at evenly spaced offsets; in this case you'll need to figure out the fence-posting when you combine the results of each of the workers.

I notice that the function preprocessing just wraps the iterable results up in a list. Is that necessary? If you can avoid ever actualizing the whole 10k-items list in memory that will likely help some.

Similarly, you're reading the whole file (http payload) as a single string. Both file handles and HTTPResponse objects will let you read just a chunk at a time; this would let you handle the subject data like a lazy iterable (similar to the way you're yielding from .finditer()). Of course how much of an advantage this is for you will depend on the underlying implementation, and you'll have to be careful about fence-posting again. This may also be a good way to break off jobs for parallelization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.