The goal of this script is to tokenize and print out all the words from the provided bible.txt file.


  • It should differentiate between 'U.K' and 'UK.' when removing extra symbols from the word.
  • It should exclude duplicates.
  • It should exclude words that contain digits.

Existing code:

#!/usr/bin/env python3

from sys import stdin
import spacy

def spacy_wordlist() -> None:
    nlp = spacy.load("en_core_web_sm")
    seen = set()

    for line in stdin:
        doc = (word for word in nlp(line.strip()) if word.is_alpha)
        for token in doc:
            if token.text not in seen:

if __name__ == "__main__":


$ neofetch --stdout 

OS: Linux Mint 21.2 x86_64 
Host: VMware Virtual Platform None 
Kernel: 5.15.0-76-generic 
CPU: Intel i5-8350U (2) @ 1.896GHz 
GPU: 00:0f.0 VMware SVGA II Adapter 
Memory: 716MiB / 1929MiB 

$ wc bible.txt
99809  821115 4432263 bible.txt

$ time ./wordlist.py < bible.txt

real    33m31.760s
user    33m9.423s
sys 0m9.407s

The execution time was considerably slow, prompting a review for optimization.

Review goal:

How can I reduce the execution time?

  • 2
    \$\begingroup\$ Are you actually printing out every word you discover? \$\endgroup\$
    – Booboo
    Jan 24 at 20:13
  • \$\begingroup\$ Why is this being done; and perhaps more to the point - why is it being done in Python? \$\endgroup\$
    – Reinderien
    Jan 24 at 20:44

3 Answers 3


right tool for the job

You called nlp(line.strip()) to tokenize the input text.

Using the power of spacy here is not appropriate. It can do many things, and you're barely taking advantage of any of them. No need to pay the cost for what we don't use.

Please see the enclosed code below. I downcase all words, and otherwise conform almost exactly to the token results your original code was producing. Running time with spacy 3.7.2 on my laptop is about 8 minutes (you had reported three or four times that, unsure why, unless you're not yet using cPython 3.11). Elapsed time without spacy is about 12 seconds, a 40x speedup. Coding my very simple algorithm in Rust would doubtless give sub-second results.

edge cases

I downcased everything, including proper nouns.

I ignored your "distinguish 'U.K' versus 'UK.'" requirement. So write a regex to notice such things, and invoke the expensive spacy tokenizer just on matching lines, preferring the cheap tokenizer for the majority of all lines scanned. At the moment the code would emit the pair "u", "k", versus just "uk"

It's worth noting that I turn . period into SPACE -- you might possibly prefer to delete them instead.

I turn - dash into SPACE, to correctly parse e.g. "standard-bearer" as two words, same as spacy does.

In the sentence, "it was his wont to take morning walks", spacy parses "wont" in a peculiar way. It emits two tokens, "wo" & "nt", as though parsing a "won't" contraction.

And "cannot" seems to be a stopword for similar reasons ("can" & "not").

modern locutions

I stopped worrying about mismatches at the end of the real corpus. Two to five millennia ago very few people spoke of filespecs or URLs; "10.zip" and the Gutenberg web address get tokenized differently from spacy's behavior, which I decided was just fine. Similarly for lawyer references to e.g. paragraph F. These things could be finessed if it's important when scanning some other corpus.


This code produces a pair of debugging log files. They are convenient for monitoring progress. But even better, they are formatted for convenient diffing, so we can verify the simple tokenizer's results are nearly identical to the reference results produced by spacy.

tokenizing algorithm

It's pretty simple. Accept a lowercase line of text, which conveniently makes a sentence's initial "The ..." look like an embedded "... and the ...".

Use str.translate() to rapidly turn punctuation into SPACE characters. Given that the example corpus always separates sentences with . period + SPACE, it might possibly make more sense to delete periods, mapping e.g. "U.K" --> "uk"

The words are now space-delimited, so it's straightforward to updeliver them, filtering out undesired words like chapter:verse annotations.

The only other value-add spacy was offering at this point was normalizing possessives, e.g. "brother's" --> "brother". The English language certainly has its irregularities, but fortunately the rule for stripping possessives is trivially implemented. And then we're done!

proper nouns

If identifying person and place names is important to your use case, you might scan the text for capitalized names that don't start a sentence: "word Name". Then re-scan, and carefully preserve such capitalized words, even when they begin a sentence.

If a name is seldom used, and it appears only at start of sentence, then we might need a POS-tagger, or go back to the full spacy parse that you had been doing.

While we're on that topic, handing spacy one line at a time works, but it's a little weird. Sending in a sentence or paragraph would be more natural, and would reveal more about each word's part-of-speech. Simply scanning for the blank line that separates paragraphs / verses would be the most convenient approach for chunking up the text.

One advantage of the current line-oriented approach is that lines are typically smaller than sentences, and after we've seen a few hundred unique words it will be commonly the case that a given line is comprised entirely of seen words. So we might do a cheap parse to identify each "boring" line, and only do an expensive spacy call on lines containing novel words. Life is full of engineering tradeoffs.

#! /usr/bin/env _TYPER_STANDARD_TRACEBACK=1 python
from collections import Counter
from operator import itemgetter
from pathlib import Path
from pprint import pp
from typing import Generator, TextIO

from spacy.language import Language
from spacy.tokens.token import Token
import spacy
import spacy.tokens
import typer

punctuation = str.maketrans(
    "              ",

def _get_simple_words(line: str) -> Generator[str, None, None]:
    for word in line.translate(punctuation).split():
        # strip possessives
        if word.endswith("'"):
            word = word[:-1]
        if word.endswith("'s"):
            word = word[:-2]

        if word.isalpha():
            yield word

def _get_spacy_tokens(nlp: Language, line: str) -> Generator[Token, None, None]:
    yield from (word for word in nlp(line.strip()) if word.is_alpha)

def spacy_wordlist(
    fin: TextIO,
    simp_out: TextIO,
    spcy_out: TextIO,
) -> None:
    nlp = spacy.load("en_core_web_sm")
    simp_seen = {"cannot"}
    spcy_seen = set()
    cnt: Counter[str] = Counter()
    dups = 0

    for line_num, line in enumerate(fin):
        line = line.lower()
        simp_out.write(f"\n{1 + line_num}\n\n")
        spcy_out.write(f"\n{1 + line_num}\n\n")

        for word in _get_simple_words(line):
            if word not in simp_seen:

        for token in _get_spacy_tokens(nlp, line):
            cnt[token.text] += 1
            if token.text in spcy_seen:
                dups += 1

    cnt = Counter({k: v for k, v in cnt.items() if v >= 3})
    pp(sorted(cnt.items(), key=itemgetter(1)))
    print(len(spcy_seen), dups)

def main(in_file: Path) -> None:
    with open(in_file) as fin:
        temp = Path("/tmp")
        simp_txt = temp / "bible_simple.txt"
        spcy_txt = temp / "bible_spacy.txt"
        with open(simp_txt, "w") as simp_out, open(spcy_txt, "w") as spcy_out:
            spacy_wordlist(fin, simp_out, spcy_out)

if __name__ == "__main__":

First spaCy will have to parse the "en_core_web_sm" file, and make sure it is valid. The result, the language model, will be in nlp, a natural language processing object.

Then you go over each and every line of the input, probably processing much more than just the words (or tokens in spaCy terms). My expectations of "natural language processing" would at least be much higher than that, and the feature set of spaCy corroborates this. Probably you are better off just using the tokenizer as you don't seem to need any of the other features. It would probably be still faster if you'd just detect the words yourself, as a big framework requires a lot of flexibility which you don't need at all.

As noted in the comments, printing all of the found words will take a lot of time, mainly depending on the speed of the terminal that the prints will go to. A lot of terminals will buffer a lot of text, so that will definitely slow down the screen printing. If you do want to print a lot of text to a terminal then why not use GPU acceleration just like a browser does? Windows Terminal should also provide GPU acceleration (most IDE's terminal views don't).

One issue with the Python set and that is that the add method does not directly indicate if the item was already in the set. This requires double hashing and verification that the items is in the set. As I expected there is a way around this by checking if the set size changes upon addition. Alternatively you can just add to the set if you're not interested in the amount of dupes. No need to check if the word is already in as that takes almost as much time as adding it in the first place...

Still, please don't expect any kind of execution time that wc brings. wc also does a word count, but it won't check for dupes. wc has been around for a long time and will likely have been optimized to the extreme; it won't take more than the I/O time (and I presume that after one test the bible.txt will be cached in RAM). And, yes, C will trump Python when it comes to these kind of "simple" actions.

  • \$\begingroup\$ This or the NLP will try and make sense of the bible... ;) Sorry, I have to leave at least one joke in. \$\endgroup\$ Jan 24 at 23:26

You can use parallel processing to take advantage of multiple CPU cores. spaCy supports processing documents in batches using nlp.pipe(), which can be combined with Python's concurrent execution features like concurrent.futures.

#!/usr/bin/env python3

import spacy
from spacy.tokens import Token
import sys

# Custom extension to handle special cases like 'U.K'
Token.set_extension('is_clean_alpha', getter=lambda token: token.text.replace('.', '').isalpha() and not any(char.isdigit() for char in token.text), force=True)

def spacy_wordlist() -> None:
    nlp = spacy.load("en_core_web_sm", disable=["parser", "ner", "lemmatizer"])
    seen = set()

    # Process the text in larger chunks instead of line-by-line
    text = sys.stdin.read()
    docs = nlp.pipe(text.split('\n'))

    for doc in docs:
        for token in doc:
            # Use the custom 'is_clean_alpha' attribute to check tokens
            if token._.is_clean_alpha and token.text not in seen:

if __name__ == "__main__":

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.