10
\$\begingroup\$

I've been trying to get more understanding and experience in the Natural Language Processing space and, to get some more practice, decided to attempt to make a simple high-level analysis of the "Industrial Society and its Future" (aka "Unabomber Manifesto").

Here is what I am trying to do:

  • download the text of the Manifesto from the Washington Post website
  • preprocess it using nltk:
    • tokenize into words
    • remove the non-alpha words
    • remove English stop words
    • lemmatize the tokens
  • count most commonly used words
  • use the TF-IDF model from gensim library to compute the most popular and most rarely used / important words

The code:

from bs4 import BeautifulSoup

from gensim.models.tfidfmodel import TfidfModel
from gensim.corpora.dictionary import Dictionary

from nltk import word_tokenize, Counter, sent_tokenize
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords

import requests


ENGLISH_STOPS = set(stopwords.words('english'))


def retrieve_text(url):
    """Downloads HTML content from a URL, parses with an HTML parser and returns text only."""
    response = requests.get(url)

    soup = BeautifulSoup(response.content, "html.parser")
    return soup.get_text()


def preprocess(text):
    """Pre-processes the text, splits into tokens that are lower-cased, filtered and lemmatized."""
    tokens = word_tokenize(text)

    # lowering
    lower_tokens = [token.lower() for token in tokens]

    # Retain alphabetic words: alpha_only
    alpha_only = [t for t in lower_tokens if t.isalpha()]

    # Remove all stop words: no_stops
    no_stops = [t for t in alpha_only if t not in ENGLISH_STOPS]

    # Lemmatize all tokens into a new list: lemmatized
    wordnet_lemmatizer = WordNetLemmatizer()
    lemmatized = [wordnet_lemmatizer.lemmatize(t) for t in no_stops]

    return lemmatized


def tf_idf(text):
    """
    Computes the TF-IDF of a text - using every sentence as a separate "document".
    Returns a list of tuples with words and weights.
    """
    texts = [preprocess(sentence) for sentence in sent_tokenize(text)]

    dictionary = Dictionary(texts)
    corpus = [dictionary.doc2bow(text) for text in texts]

    tfidf = TfidfModel(corpus)
    corpus_tfidf = tfidf[corpus]

    tfidf_weights = {dictionary.get(id): value
                     for doc in corpus_tfidf
                     for id, value in doc}
    sorted_tfidf_weights = sorted(tfidf_weights.items(), key=lambda w: w[1])

    return sorted_tfidf_weights


if __name__ == '__main__':
    content = retrieve_text("http://www.washingtonpost.com/wp-srv/national/longterm/unabomber/manifesto.text.htm")

    tokens = preprocess(content)
    token_counter = Counter(tokens)

    most_common = token_counter.most_common(10)
    tf_idf_results = tf_idf(content)
    popular_terms, rare_terms = tf_idf_results[:10], tf_idf_results[-10:]

    # print out words only (without counts and weights)
    keys = lambda x: next(zip(*x))
    print(f"Most common words: {keys(most_common)}")
    print(f"Most popular terms: {keys(popular_terms)}")
    print(f"Most unique/important terms: {keys(rare_terms)}")

Prints:

Most common words: ('society', 'system', 'people', 'power', 'would', 'one', 'human', 'technology', 'leftist', 'need')
Most popular terms: ('society', 'people', 'freedom', 'whole', 'human', 'lead', 'system', 'necessity', 'process', 'use')
Most unique/important terms: ('license', 'simplification', 'personnel', 'carried', 'crossroad', 'eminent', 'exactly', 'paramount', 'danger', 'virtue')

I am still learning and would like my code to be reviewed. Would appreciate any feedback about the code quality, performance or any other possible improvements.

Also, I am not 100% sure the TF-IDF model is applied in the most appropriate way. Currently, I am treating each sentence as a separate document. Is this a good approach to detect the most "important"/"valuable" terms/words, or should I use other articles/posts as documents for the corpus?

\$\endgroup\$
3
\$\begingroup\$

First, I will start with PEP 8 specifications. The PEP 8 analysis shows the following:

E501:17:80:line too long (94 > 79 characters)
E501:25:80:line too long (99 > 79 characters)
E501:46:80:line too long (82 > 79 characters)
E501:66:80:line too long (114 > 79 characters)
E731:76:5:do not assign a lambda expression, use a def
W292:79:62:no newline at end of file

If I am not incorrect, one of the things that you are trying is to calculate the frequency distribution of words, namely the most common words and most rare words. I tried to re-write your code like viz.

from bs4 import BeautifulSoup
from nltk import FreqDist, re

import requests

def retrieve_text(url):
    """Downloads HTML content from a URL, parses with an HTML parser and returns text only."""
    response = requests.get(url)

    soup = BeautifulSoup(response.content, "html.parser")
    return soup.get_text()


if __name__ == '__main__':

    content = retrieve_text("http://www.washingtonpost.com/wp-srv/national/longterm/unabomber/manifesto.text.htm")
    wordList = re.sub("[^\w]", " ", content).split()
    fdist = FreqDist(wordList)

    # print out words only (without counts and weights)
    print(fdist.most_common(20))

As you see in the output, I am obviously not skipping connectorwords, like the, of etc.:

[('the', 1735), ('of', 1251), ('to', 1078), ('a', 736), ('and', 719), ('that', 657), ('is', 611), ('in', 546), ('be', 412), ('for', 351), ('it', 319), ('or', 290), ('are', 281), ('have', 246), ('society', 239), ('not', 235), ('will', 233), ('as', 230), ('by', 229), ('they', 224)]

but, the execution was much faster and I did not have to spend a huge time downloading the gensim library. Off course if you need something as sophisticated as gensim, I have no substitute, else for frequency, this maybe much faster and you can easily write a method to remove connector words.

\$\endgroup\$
2
+100
\$\begingroup\$

Caching

One important step would be to download the page only if it hasn't been downloaded before. This way, the script might run a bit faster, it also works without Internet connection and your name doesn't appear on a "No Fly List" because you've downloaded the Unabomber manifesto a hundred times.

from gensim.models.tfidfmodel import TfidfModel
from gensim.corpora.dictionary import Dictionary

from nltk import word_tokenize, Counter, sent_tokenize
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords

import os


ENGLISH_STOPS = set(stopwords.words('english'))


def retrieve_text(url):
    """Downloads HTML content from a URL, parses with an HTML parser and returns text only."""
    html_filename = os.path.basename(url)
    txt_filename = os.path.splitext(html_filename)[0] + '.txt'

    if not os.path.exists(txt_filename):
        if not os.path.exists(html_filename):
            import requests
            response = requests.get(url)
            with open(html_filename, 'wb') as html:
                html.write(response.content)

        from bs4 import BeautifulSoup
        with open(html_filename, 'rb') as html:
            with open(txt_filename, 'w') as txt:
                soup = BeautifulSoup(html.read(), "html.parser")
                txt.write(soup.get_text())

    with open(txt_filename) as txt:
        return txt.read()

Generators

You don't actually need lower_tokens, alpha_only or no_stops as lists. They could be generators. This way, you keep the comments and the variable names, but you only iterate once over tokens and don't create unneeded lists.

# lowering
lower_tokens = (token.lower() for token in tokens)

# Retain alphabetic words: alpha_only
alpha_only = (t for t in lower_tokens if t.isalpha())

# Remove all stop words: no_stops
no_stops = (t for t in alpha_only if t not in ENGLISH_STOPS)
\$\endgroup\$
1
\$\begingroup\$

One obvious improvement to performance can be made in preprocess. Currently you are using 4 list comprehensions to do your processing. This means you are storing 4 copies of the data, and making 4 passes through the data. By combining steps, you should see a noticeable performance boost. The following should be a fair bit faster, especially for longer text.

def preprocess(text):
    """Pre-processes the text, splits into tokens that are lower-cased, filtered and lemmatized."""

    # lowering
    # Retain alphabetic words: alpha_only
    # Remove all stop words: no_stops
    tokens = (token.lower() for token in word_tokenize(text)
                            if t.isalpha()
                            and t.lower() not in ENGLISH_STOPS)

    # Lemmatize all tokens into a new list: lemmatized
    wordnet_lemmatizer = WordNetLemmatizer()
    return = [wordnet_lemmatizer.lemmatize(t) for t in tokens]

Other than that your code looks pretty good.

\$\endgroup\$
  • 1
    \$\begingroup\$ Have you actually timed this? As this looks to me to be reducing memory usage, where performance is roughly the same. \$\endgroup\$ – Peilonrayz Dec 5 '17 at 16:37
  • \$\begingroup\$ In general reduced memory leads to better performance mainly when working with large amounts of data. With small files it should be about the same. \$\endgroup\$ – Oscar Smith Dec 5 '17 at 18:17
  • \$\begingroup\$ Performance and memory usage are usually trade-offs, and I can only see a marginal speedup from your change, at best. Heck due to two token.lower()s, it could be tons worse. \$\endgroup\$ – Peilonrayz Dec 5 '17 at 18:30
  • \$\begingroup\$ Since there is no speed penalty, and it uses less memory, I'd still call it a win. \$\endgroup\$ – Oscar Smith Dec 5 '17 at 18:33
  • 2
    \$\begingroup\$ Nowhere do you mention 'less memory' in you answer, you only talk about speed. \$\endgroup\$ – Peilonrayz Dec 5 '17 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.