# Counting and scoring two- and three-word phrases

I am writing a function to take a list of words and score 2 and 3 word phrases based on occurrence while removing any phrase occurrences with a score of less than 3.

As someone kinda new to python, i'm wondering if there is a better way to write this to do the same thing (the code works)

The resulting dictionary looks something like this...

{ ( "lorem ipsum", 4 ), ("quick brown fox", 3), (...) }


## -words- is a list of words
## -phrases- is the resulting dictionary with the phrase as a key and score as a value

words = ("lorem","ipsum","lorem","ipsum","quick","brown","fox")
phrases = dict()

for i, w in enumerate(words):

w = re.sub('[^0-9a-zA-Z]+', '', w).lower()

try:
w1 = " " + re.sub('[^0-9a-zA-Z]+', '', words[ i + 1 ]).lower()
except:
w1 = ""

try:
w2 = " " + re.sub('[^0-9a-zA-Z]+', '', words[ i + 2 ]).lower()
except:
w2 = ""

if w!="" and not w.isdigit() and len(w)>1:

p = w + w1
p2 = w + w1 + w2

try:
phrases[p] = phrases[p] + 1
except:
phrases[p] = 1

try:
phrases[p2] = phrases[p2] + 1
except:
phrases[p2] = 1

phrases = { k:v for k, v in phrases.items() if v > 2 }

rc = sorted(phrases.items(), key=lambda x: x, reverse = True)

• Please clarify what you mean by "The resulting dictionary looks something like this...". What exactly is the expected result for the sample word list in the question? If your example doesn't match the code, then how can we tell what the intended behavior is? – 200_success Apr 20 '17 at 3:41

So I redid your main loop to make it more pythonic. The completed inner loop is below. But first let us discuss some of the changes.

If space allows, build intermediate data structure to ease later processing

So this bit of code recasts the word list into a single list that is more readily processed in the main loop.

words_x = [re.sub('[^0-9a-zA-Z]+', '', w) for w in words] + ['', '']


It does:

1. Add some padding at the end for end of list processing. I wasn't sure that this was actually accurate to the described algorithm, but it does emulate the existing behavior.

2. Perform the regex substitution only once.

If space does not allow building another data structure, the itertools module has various tools that may allow efficient iterating over a data structures, and is often worth investigating.

zip() allows processing more than one list at a time

So I restructured the loop to look like:

for w, w1, w2 in zip(words_x, words_x[1:], words_x[2:]):


This iterates through without the need for a list index and gets three adjacent words each iteration.

zip() works by iterating through each of its arguments, and presenting the first element of each list the first time it is called, and then the second element of each list, and then... And by calling it with the same list but starting at the 1st, 2nd and 3rd elements respectively, we can get three adjacent elements each iteration.

Use collections.Counter

So python has a nifty data structure that acts a lot like a dictionary but is used for counting. It is oddly enough called Counter.

Counter allows replacing:

try:
phrases[p] = phrases[p] + 1
except:
phrases[p] = 1


with:

phrases[w] += 1


Code:

from collections import Counter

words = ("lorem", "ipsum", "lorem", "ipsum", "quick", "brown", "fox")
words_x = [re.sub('[^0-9a-zA-Z]+', '', w) for w in words] + ['', '']

phrases = Counter()
for w, w1, w2 in zip(words_x, words_x[1:], words_x[2:]):

if len(w) > 1 and not w.isdigit():
if w1:
w += ' ' + w1
phrases[w] += 1
if w2:
w += ' ' + w2
phrases[w] += 1


### Bugs?:

Code walks off the end of the list:

The current code walks off the end of the list. The effect of this can be seen by the fact that the single word 'fox' ends up in the phrases list. The current code looks for any phrase that exists more than once, so this word is eventually dropped, but...

At the end of the list the last two word phrase double counted as a three word phrase

In the current code, brown fox is counted twice.

The code below fixes both terminal condition problems.

Revised Code:

from collections import Counter

words = ("lorem", "ipsum", "lorem", "ipsum", "quick", "brown", "fox")
REMOVE_EXTRA = re.compile(r'[^0-9a-zA-Z]+')
words_x = [REMOVE_EXTRA.sub('', w) for w in words] + ['']

phrases = Counter()
for w, w1, w2 in zip(words_x, words_x[1:], words_x[2:]):

if len(w) > 1 and not w.isdigit():
if w1:
w += ' ' + w1
phrases[w] += 1
if w2:
# only count as three word phrase if w2 is not blank
phrases[w + ' ' + w2] += 1