Utility functions for processing tweets or texts

Question

I have 4 functions that process texts or tweepy.models.Status object (this object stores text information, author information, retweets & likes information of a specific tweet).

The first function is filter_unique, which is used to exclude duplicate tweets from a list of tweets. The result is a generator.

The function clean_text, is used to remove punctuations, undefined characters, and whitespaces from a string.

The function split_texts has two inputs, the list of tweets/strings and the mode of the splits (naive=True means it only use text.split(), if set to False then it will clean the text first before splitting)

The function total_rts calculates the total number of retweets. string_inclusion='word' means that it will calculate the total retweets of tweets that contain 'word'.

Question: How to make this module more efficient, and perhaps advanced?

We can use the sample tweets testfile.npy which contains objects tweepy.models.Status.

---

import tweepy
import string

def filter_unique(tweets):

    uniques = set()

    for tweet in tweets:

        if not isinstance(tweet, tweepy.Status):
            raise TypeError('Each element must be of tweepy.Status object')

        try:
            tweet = tweet.retweeted_status
        except:
            pass

        if tweet.id not in uniques:

            uniques.add(tweet.id)

            yield tweet

def clean_text(text):

    punct = string.punctuation
    printable = string.printable
    whitespace = string.whitespace

    table = text.maketrans({key: None for key in printable})
    undef = set(text.translate(table))

    table = text.maketrans({key: None for key in undef})
    removed_undef = text.translate(table)

    table = text.maketrans({key: None for key in punct})
    cleaned = removed_undef.translate(table)

    table = text.maketrans({key: ' ' for key in whitespace})
    cleaned = cleaned.translate(table)

    return cleaned

def split_texts(texts, naive):

    if naive:
        for text in texts:
            yield text.split()
    else:
        for text in texts:
            yield clean_text(text).split()

def total_rts(tweets, string_inclusion = False, naive = True):

    result = 0

    if not string_inclusion :

        result = sum([tweet.retweet_count for tweet in tweets]);

    else:

        if naive:
            try:
                result = sum([tweet.retweet_count for tweet in tweets if string_inclusion in tweet.full_text.split()])
            except:
                result = sum([tweet.retweet_count for tweet in tweets if string_inclusion in tweet.text.split()])
        else:
            try:
                result = sum([tweet.retweet_count for tweet in tweets if string_inclusion in clean_text(tweet.full_text).split()])
            except:
                result = sum([tweet.retweet_count for tweet in tweets if string_inclusion in clean_text(tweet.text).split()])

    return result

Answer 1

You have two bare except. Avoid doing that so it is:

1. easier to understand what kind of errors you are expecting (it seems both try to catch AttributeErrors but it is not entirely clear);
2. less error prone as you won't act on exceptions you didn't anticipate (such as MemoryError or KeyboardInterrupt).

---

The way you handle the naive argument in split_text and total_rts makes for duplicated code. You can take advantage of generator functions or generators to prepare your data depending on the argument and then write the computation once. In your case, map can go a long way into preparing your data before processing without actually computing anything before the last moment:

import operator


def split_texts(texts, naive):
    if not naive:
        texts = map(clean_text, texts)
    yield from map(str.split, texts)


def total_retweets(tweets, string_inclusion=False, naive=True):
    if not string_inclusion:
        return sum(tweet.retweet_count for tweet in tweets)

    def filtered_retweets(text_attribute):
        texts = map(operator.attrgetter(text_attribute), tweets)
        if not naive:
            texts = map(clean_text, texts)
        texts = map(str.split, texts)

        return sum(tweet.retweet_count for tweet, text in zip(tweets, texts) if string_inclusion in text)

    try:
        return filtered_retweets('full_text')
    except AttributeError:
        return filtered_retweets('text')

Answer 2

A few obligatory PEP-8 rules:

• Imports should be grouped in the following order:

  1. standard library imports
  2. related third party imports
  3. local application/library specific imports
• Use blank lines in functions, sparingly, to indicate logical sections.
• Two blank lines between the import statements and other code.
• Two blank lines between each function.
• Don't use spaces around the = sign when used to indicate a keyword argument or a default parameter value.

If this is written for python-3.x, I'd also suggest going for Type Hinting.

---

In the split_texts definition, I'd suggest having naive=False as the default (or naive=True) depending on the more obvious choice of the two.

---

In the function total_rts (could be renamed to total_retweets), you can avoid one entire level of nesting by returning early in the first if not string_inclusion clause.

---

The function clean_text makes several transition tables, which could be extracted out of the function and made a constant. More importantly, what I understood from the definition, I think you're (in order):

1. cleaning the text of all printable characters
2. cleaning the text of all characters leftover from step 1
3. cleaning all punctuation from result of step 2
4. replacing all kinds of whitespace characters with a single ' ' in the result of step 3.

You can define global transition tables as follows:

WHITESPACE = ' ' * len(string.whitespace)
CLEAN_TRANSLATION = str.maketrans(string.whitespace, WHITESPACE, string.punctuation)

Alternatively, you can take the CLEAN_TRANSLATION inside the clean_text function and use the undefined non-printable characters in concatenation to string.punctuation. So, the function becomes:

def clean_text(text):
    undefined_chars = text.translate(
        str.maketrans({c: None for c in string.printable}))
    cleanup_translation = str.maketrans(
        string.whitespace, WHITESPACE, string.punctuation + undefined_chars)
    return text.translate(cleanup_translation)

---

And, most importantly, the description for each function that you've provided in the question, could be converted to a docstring for each of them.