# Tokenizing a large document

I'm currently trying to process a corpus of a million patent text files, which contain about 10k non-unique words on average. My current data pipeline works as follows:

1. Load the patent texts as strings from a local sqlite database table
2. Tokenize¹ each document and save the results in a new table
3. Create a dictionary containing all unique words in the corpus
4. Train an tfidf model with the tokenized documents

¹Tokenization means taking a document text (string) as input and returning a list containing every word in the document (duplicates allowed). Words tend to be separated by spaces, special characters, numbers etc., the regex in my code has served quite well for this purpose.

In my data pipeline, I identified the tokenize function as my bottleneck, the relevant part is provided in my MWE below:

import re
import urllib.request
import time

url='https://raw.githubusercontent.com/mxw/grmr/master/src/finaltests/bible.txt'

PAT_ALPHABETIC = re.compile(r'[^\W\d]+')

def tokenize(text):
matches=PAT_ALPHABETIC.finditer(text)
for match in matches:
yield match.group()

def preprocessing(doc):
tokens = [token for token in tokenize(doc)]

start_time = time.time()
preprocessing(doc)
print("--- %s seconds ---" % (time.time() - start_time))

• Is that a vocabulary of 10k distinct words per document (not mean, esp. with stemming), or a mere 10 k words per text file - say, about 100 k characters? I would not consider the latter very big, but a large amount again may amount to a considerable total. Apr 8, 2020 at 22:50
• It's the latter one. I'm processing roughly a million patent texts which tend to have 10-20k (non-distinct) words per text on average. Due to the technical nature of patents, the number of distinct words also gets very large when considering the whole corpus. Apr 9, 2020 at 8:59
• I'd expect insight into the overall problem more promising with more code context provided. Can you present, in your question (where information like patent texts would be placed advantageously), method and measurement results that let you [identify] the tokenize function as [the] bottleneck? Apr 9, 2020 at 10:12
• Hip-shot: time WHITE_DIGITS = re.compile(r'[\W\d]+') WHITE_DIGITS.split(doc) Apr 9, 2020 at 10:24
– Mast
Apr 9, 2020 at 11:27

Some small performance can be gained by directly yielding from the iterator instead of having a for loop do it, using the yield from keywords, and using list() instead of a list comprehension:

def tokenize2(text):
yield from PAT_ALPHABETIC.finditer(text)

def preprocessing2(doc):
return list(tokenize2(doc))


For the given example document this gives about a 15% speed-up:

In [15]: %timeit preprocessing(doc)
335 ms ± 2.29 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [16]: %timeit preprocessing2(doc)
287 ms ± 2.79 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)


Slightly faster yet is not even having the preprocessing function and directly returning all tokens (this avoids one function call and lets the re do its best):

def tokenize3(text):
return PAT_ALPHABETIC.findall(text)


This is about 35% faster then the code in the OP:

In [21]: %timeit tokenize3(doc)
217 ms ± 1.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)


I notice that the function preprocessing just wraps the iterable results up in a list. Is that necessary? If you can avoid ever actualizing the whole 10k-items list in memory that will likely help some.
Similarly, you're reading the whole file (http payload) as a single string. Both file handles and HTTPResponse objects will let you read just a chunk at a time; this would let you handle the subject data like a lazy iterable (similar to the way you're yielding from .finditer()). Of course how much of an advantage this is for you will depend on the underlying implementation, and you'll have to be careful about fence-posting again. This may also be a good way to break off jobs for parallelization.