Word frequency counter

Question

I recently took a self-assessment question to assess my Python ability for an online class. The problem was to return the frequency of a word occurring, as part of a tuple.

Implement a function count_words() in Python that takes as input a string word_string and a number number_of_words, and returns the n most frequently-occurring words in word_string. The return value should be a list of tuples - the top n words paired with their respective counts [(, ), (, ), ...], sorted in descending count order.

You can assume that all input will be in lowercase and that there will be no punctuations or other characters (only letters and single separating spaces). In case of a tie (equal count), order the tied words alphabetically.

E.g.: print count_words("this is an example sentence with a repeated word example",3) Output: [('example', 2), ('a', 1), ('an', 1)]

def count_words(word_string, number_of_words):
  """
  take in a word string and return a tuple of the
  most frequently counted words

  word_string = "This is an example sentence with a repeated word example",
  number_of_words = 3
  return [('example', 2), ('This', 1), ('a', 1)]
  """
  word_array = word_string.split(' ')
  word_occurence_array = []
  for word in word_array:
    if word in word_string:
      occurence_count = word_array.count(word)
      word_occurence_array.append((word, occurence_count))
    else:
      # no occurences, count = 0
      word_occurence_array.append((word, 0))

  # dedupe
  word_occurence_array = list(set(word_occurence_array))

  # reorder
  # can also pass, reverse=True, but cannot apply `-` to string
  word_occurence_array.sort(key=lambda tup: (-tup[1], tup[0]))

  # only return the Nth number of pairs
  return word_occurence_array[:number_of_words]

You can then call this function:

count_words(word_string="this is an example sentence with a repeated word example", number_of_words=3)

which returns [('example', 2), ('a', 1), ('an', 1)]

I found the process of tuple sorting, quite tricky, and achieved it using word_occurence_array.sort(key=lambda tup: (-tup[1], tup[0])). I was wondering if there are any other improvements that I can make to my overall code.

---

I hope this is a reasonable question - I've tweaked the description and example so that I hope it's not too easily identifiable.

Answer 1

The suggestion made by 200_success is a good one if you don't care about the returned values in the case of a tie, however the question seems to indicate that after sorting by count, you should sort alphabetically. You'll need to add post-processing to that with a Counter (or any mapping). You could also do this with a collections.defaultdict.

from collections import defaultdict, Counter

def count_words2(word_string, number_of_words):
  words = word_string.split()
  word_dict = defaultdict(int)
  for word in words:
    word_dict[word] += 1

  return sorted(word_dict.iteritems(), key=lambda tup: (-tup[1], tup[0]))[:number_of_words]

def count_words3(word_string, number_of_words):    
  words = word_string.split()
  word_dict = Counter(words)
  return sorted(word_dict.iteritems(), key=lambda tup:(-tup[1], tup[0]))[:number_of_words]

My original answer was a bit hasty/knee-jerk reaction to the suggestion to use most_common, which gives no guarantees about ordering in the case of ties, nor can it be passed a function to handle the sorting. You can still use a Counter, you just can't use most_common without somewhat more complex post-processing. As seen above, you should be able to use any mapping to actually get the frequency table, as the post-processing step will be the same. Given the lower complexity of Counter, that is still probably the best solution.

As suggested by Mathias Ettinger in the comments, you could also do something like this

class OrderedCounter(Counter, OrderedDict): pass

def count_words4(word_string, n):

  words = OrderedCounter(sorted(word_string.split()))
  return words.most_common(n)

In general I prefer to avoid multiple inheritance unless it is clearly a much cleaner and simpler solution than anything else - I don't think that is accurate in this case, but you might decide that it works for you.

Answer 2

This is exactly the kind of task that should be handled using collections.Counter, which offers a most_common() method to extract the results.

Answer 3

Another thing I'd like to comment on (I only need to look at the start), so I'm copy/pasting that.

def count_words(word_string, number_of_words):
  """
  take in a word string and return a tuple of the
  most frequently counted words

  word_string = "This is an example sentence with a repeated word example",
  number_of_words = 3
  return [('example', 2), ('This', 1), ('a', 1)]
  """

First of all, the indent is 'wrong' (indenting in Python is traditionally 4 spaces).

Second, there's a traditional format for writing examples. It's called doctests. It looks just like the REPL.

def count_words(word_string, number_of_words):
    """
    take in a word string and return a tuple of the
    most frequently counted words.

    >>> word_string = "This is an example sentence with a repeated word example"
    >>> number_of_words = 3
    >>> count_words(word_string, number_of_words)
    [('example', 2), ('a', 1), ('an', 1)]
    """
    ... rest of code here ...

Then, using python -m doctest -v modulename.py, it will run your the parts prefixed by >>>, and check the output matches. I fixed the example, because the tie-resolution was not done properly in your example.