Retrieve the shared words from files

Question

I have two files of dip.md and tp.md, which copied respectively from Think Python and Dive Into Python 3

By comparing them, I retrieved the shared words by the following procedures:

1. Read contents from files

   def read(filename):
       with open(filename) as file:
           content = file.read()
       return content
   dip = read('dip.md')
   tp = read('tp.md')
   
   In [49]: dip
   Out[49]: '- -  \n\n    You are here:  •\n\n    # Dive Into Python 3\n\n    Dive Into Python 3 covers Python 3 and its differences from Python 2. Compared to [Dive Into Python](http://diveintopython.net/), it’s about 20% revised and 80% new material. The book is now complete, but [feedback is always welcome](http://www.diveintopython3.net/about.html).\n\n
   ...
   In [50]: tp
   Out[50]: 'Think Python: How to Think Like a Computer Scientist\n\nAllen B. Downey\n\n2nd Edition, Version 2.2.23\n\nPreface\nThe strange history of this book\nAcknowledgments\nContributor List\nThe way of the program\nWhat is a program?\nRunning Python\nThe first program\nArithmetic operators\nValues and types\nFormal and natural languages\nDebugging\nGlossary\nExercises\nVariables, expressions and statements\nAssignment statements\nVariable names\nExpressions and statements\nScript mode\nOrder of operations\nString operations\nComments\nDebugging\nGlossary\nExercises\nFunctions\nFunction calls
   

2. Clear data using string.punctuation

   def clear_data(cont):
       for data in cont:
           if data in punctuation:
               cont = cont.replace(data, ' ')
       cont = cont.replace('\n', ' ')
       cont = cont.lower()
       return cont
   dip_cont = clear_data(dip)
   tp_cont = clear_data(tip)
   

3. Retrieve the qualified list

   def get_qualified_list(cont):
       cont_list = cont.split(' ')
       qualified_list = [i for i in cont_list if i.isalpha()]
       return qualified_list
   dip_list = get_qualified_list(dip_cont)
   tp_list = get_qualified_list(tp_cont)
   

4. Get their intersection

   In [51]:  print(set(dip_list) & set(tp_list))
   {'is', 'study', 'Refactoring', 'with', 'and', 'the', 'first', 'Expressions', 'book', 'as', 'comprehensions', 'of', 'Strings', 'program', 'names', 'expressions', 'Case', 'to', 'Files', 'Classes', 'Objects', 'method', 'are', 'Python', 'The', 'in', 'new', 'strings'}
   

This solution is a bit too much for such a task which can be easily handled on 'MS Word'.

How can such a task be accomplished in an easy way?

Answer 1

This solution is a bit too much for such a task

Not even close. Welcome to the Natural Language Processing space :)

In order to properly compare the English texts we at least need to apply the following things:

• make the comparison case-insensitive
• remove punctuation and retain alphabetic words only
• remove stop words
• lemmatize words

We can make use of the awesome nltk Python library to help us tokenize, remove the stop words and lemmatize. Here is something more or less generic that works for HTML documents (you can modify the "download content" part if you are working with markdown or other document types):

from bs4 import BeautifulSoup
from nltk import word_tokenize
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords
import requests


ENGLISH_STOPS = set(stopwords.words('english'))


def download_content(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html.parser")

    return soup.get_text()


def preprocess(text):
    tokens = word_tokenize(text)

    # lowering
    lower_tokens = [token.lower() for token in tokens]

    # Retain alphabetic words: alpha_only
    alpha_only = [t for t in lower_tokens if t.isalpha()]

    # Remove all stop words: no_stops
    no_stops = [t for t in alpha_only if t not in ENGLISH_STOPS]

    # Lemmatize all tokens into a new list: lemmatized
    wordnet_lemmatizer = WordNetLemmatizer()
    lemmatized = [wordnet_lemmatizer.lemmatize(t) for t in no_stops]

    return set(lemmatized)


def compare_documents(*urls):
    contents = [download_content(url) for url in urls]

    return set.intersection(*[set(preprocess(content)) for content in contents])


if __name__ == '__main__':
    print(compare_documents(*[
        "http://greenteapress.com/thinkpython2/html/index.html",
        "http://www.diveintopython3.net"
    ]))

Prints:

{'object', 'generator', 'string', 'first', 'study', 'book', 'name', 'refactoring', 'new', 'case', 'program', 'expression', 'file', 'python', 'method', 'class', 'common', 'comprehension'}

These are, of course, like in your example, not the full books that were compared - just the contents pages of the two.

By the way, here are the common words between your question and my answer (computed using the proposed code):

{
    'task', 'python', 'content', 'following', 'two', 'welcome',
    'natural', 'much', 'return', 'new', 'list', 'book', 'print', 'like',
    'compared', 'def', 'word', 'http', 'punctuation', 'solution', 'bit', 'set'
}