I'm currently trying to process a corpus of a million patent text files, which contain about 10k non-unique words on average. My current data pipeline works as follows:
- Load the patent texts as strings from a local sqlite database table
- Tokenize¹ each document and save the results in a new table
- Create a dictionary containing all unique words in the corpus
- Train an tfidf model with the tokenized documents
¹Tokenization means taking a document text (string) as input and returning a list containing every word in the document (duplicates allowed). Words tend to be separated by spaces, special characters, numbers etc., the regex in my code has served quite well for this purpose.
In my data pipeline, I identified the tokenize function as my bottleneck, the relevant part is provided in my MWE below:
import re import urllib.request import time url='https://raw.githubusercontent.com/mxw/grmr/master/src/finaltests/bible.txt' doc=urllib.request.urlopen(url).read().decode('utf-8') PAT_ALPHABETIC = re.compile(r'[^\W\d]+') def tokenize(text): matches=PAT_ALPHABETIC.finditer(text) for match in matches: yield match.group() def preprocessing(doc): tokens = [token for token in tokenize(doc)] return tokens start_time = time.time() preprocessing(doc) print("--- %s seconds ---" % (time.time() - start_time))