# Check words in documents

Suppose you have two lists:

• One is a "dictionary", a list of words
• The other is "documents", a list of documents (which in turn, are strings)

I would check whether a dictionary's word is within a document.

# DOCUMENTS
doc1 = "My sister likes to have sugar, but not my father."
doc2 = "Doctors suggest that driving may cause increased stress."

documents = [doc1, doc2]

# DICTIONARY
dictionary = ["doctors", "sister", "like"]

# CHECK DOCUMENT WORDS WITHIN DICTIONARY
for word in dictionary:
for doc in documents:
if word in doc:
print "\n\n\nWord:",
print word,
print "in document:",
print doc


What's a more "pythonic" way?

• How dictionary and documents variables are defined? Could you demonstrate how your program works on a sample input and what output it produces? Thanks. – alecxe Mar 13 '17 at 19:13
• What are you trying to accomplish? What if the word 'car' is in the dictionary, but one of the documents talks about 'carpets'? Is that an intended match? – 200_success Mar 13 '17 at 19:14
• I VTC this as broken because the if word in doc: doesn't check for a word within a document because of the reason 200_success mentioned above :) Due to this, the output will be different from what OP asked (e.g: 'me' in 'melt mess merry' which returns True even though me isn't actually there) – Grajdeanu Alex. Mar 13 '17 at 19:42
• Thank you for your comments and sorry for this late answer! This is a very dumb code: it directly matches the words within the dictionary with the word in documents. I added the definition of variables as asked by @alecxe. I hope the post could be reopened. – Gabrer Mar 21 '17 at 11:34
• You say in your description that " 'documents' [is] a list of documents (which in turn, are lists of words)". However, in your code each document is a string, instead of a list of words. If it actually were as you claimed, this would avoid the problem pointed out by @200_success. – Graipher Mar 21 '17 at 12:41

If your documents were actually lists of words, like your description said at one point, this would be way more efficient. But we can of course just make them, by using:

doc1 = "My sister likes to have sugar, but not my father.".split()
doc2 = "Doctors suggest that driving may cause increased stress.".split()


This has the problem that the punctuation gets in the way, so we need to remove it first, with one of the approaches from here (I chose the fastest one):

import string
doc1 = "My sister likes to have sugar, but not my father.".translate(None, string.punctuation).split()
doc2 = "Doctors suggest that driving may cause increased stress.".translate(None, string.punctuation).split()


Ok, now that they are lists, we make them into sets, so word in doc becomes $\mathcal{O}(1)$ instead of $\mathcal{O}(n)$:

documents = map(set, [doc1, doc2])

# DICTIONARY
dictionary = ["doctors", "sister", "like"]

for word in dictionary:
for doc in documents:
if word in doc:
print "\n\n\nWord:",
print word,
print "in document:",
print doc


This should speed this up considerable. The most speed-up will be gained if the document strings are quite long.

The output of this is now not really understandable anymore:

Word: sister in document: set(['sister', 'my', 'father', 'but', 'sugar', 'to', 'likes', 'have', 'not', 'My'])


We can get around that by carrying around both the set and the string. I also made the splitting into words a function and used a if __name__ == "__main__" guard to ensure this code is only run when calling it via e.g. python script.py, but not when doing from script import get_words.

import string

def get_words(text):
return set(text.translate(None, string.punctuation).split())

if __name__ == "__main__":
documents = ["My sister likes to have sugar, but not my father.",
"Doctors suggest that driving may cause increased stress."]

words_in_documents = map(get_words, documents)

for word in ["doctors", "sister", "like"]:
for doc, words in zip(documents, words_in_documents):
if word in words:
print "\n\n\nWord: {} in document: {}".format(word, doc)