5
\$\begingroup\$

Here is the code I have written to create an inverted index dictionary for a set of documents:

inv_indx = {i:[] for i in corpus_dict}
for word in corpus_dict:
    for i in range(len(docs)):
        if word in docs[i]:
            inv_indx[word].append(i)

docs is a list of sets of the words in various documents:

[{'once','upon','a','time',...},{'lorum','ipsum','time'...},...]

corpus_dict is a set of all the words that appear in any of the documents:

{'once','upon','a','time','lorum','ipsum',...}

inv_index becomes a dictionary with each word in the corpus_dict as a key for a list of the document ids that contain that word:

{'once':[0],'time':[0,1],...}

The problem is this becomes very slow if the number of documents gets too big. How can I make this code more efficient?

\$\endgroup\$
4
\$\begingroup\$

Suggestions

  • You check all words in all documents. Try iterate only over docs (without unnecessary checks)
  • instead of create empty inv_indx = {i:[] for i in corpus_dict} you can use defaultdict

Code

from collections import defaultdict

inv_indx = defaultdict(list)
for idx, text in enumerate(docs):
    for word in text:
        inv_indx[word].append(idx)
\$\endgroup\$
  • \$\begingroup\$ Wow that was so much faster. The defaultdict doesn't seem to have much impact on the perfomance though so I'll leave that out \$\endgroup\$ – Joe Mar 6 '18 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.