# Word/expression list frequency distribution

I have another Python issue, creating a frequency distribution out of a text matching a predefined wordlist. In fact, I work with more than 100,000 text files (each consisting of about 15,000 words), which I want to inread and accordingly match against a word/expression list (vocabulary_dict) of about 6,000 entries. The result should be a dictionary of all that entries attached with their respective frequency.

Here is what I am currently doing:

sample text = "As the prices of U.S. homes started to falter, doubts arose throughout the global financial system. Banks became weaker, private credit markets stopped functioning, and by the end of the year it was clear that the world banks had sunk into a global recession."

vocabulary_dict = ['prices', 'banks', 'world banks', 'private credit marktes', 'recession', 'global recession']

def list_textfiles(directory):
# Creates a list of all files stored in DIRECTORY ending on '.txt'
textfiles = []
for filename in listdir(directory):
if filename.endswith(".txt"):
textfiles.append(directory + "/" + filename)
return textfiles

for filename in list_textfiles(directory):
# inread each report as textfile, match tokenized text with predefined wordlist and count number of occurences of each element of that wordlist
splitted = nltk.word_tokenize(sample_text)
c = Counter()
c.update(splitted)
outfile = open(filename[:-4] + '_output' + '.txt', mode = 'w')
string = str(filename) # write certain part of filename to outfile
string_print = string[string.rfind('/')+1:string.find('-')] + ':' + string[-6:-4] + '.' + string[-8:-6] + '.' + string[-12:-8]
for k in sorted(vocabulary_dict):
# recognize if wordlist element consist of one or more tokens, accordingly find matching token length in text file (report)
spl = k.split()
ln = len(spl)
if ln > 1:
if re.findall(r'\b{0}\b'.format(k),sample_text):
vocabulary_dict[k.lower()] += 1
elif k in sample_text.split():
vocabulary_dict[k.lower()] += c[k]
outfile.write(string_print + '\n')
# line wise write each entry of the dictionary to the corresponding outputfile including comapany name, fiscal year end and tabulated frequency distribution
for key, value in sorted( vocabulary_dict.items() ):
outfile.write( str(key) + '\t' + str(value) + '\n' )
outfile.close()

# Output accoring to the above stated example should be in the form:
"selected part of filename (=string1)"
'prices' 1
'banks' 2
'world banks' 1
'private credit marktes' 1
'recession' 1
'global recession' 1


The code works quite well, but I still think there is room for optimization, since the time to process a textfile is ca. 1 minute.

Is there a way to make the matching of text to the word/expression list faster?

• Did you actually run the code you posted? There is a typo. – 200_success Feb 8 '15 at 15:15

One obvious improvement: you call elif k in sample_text.split(): inside the loop over sorted(vocabulary_dict), so this operation happens multiple times. Instead, you could create a set (which has more efficient membership tests) outside the loop, so it only happens once:

samples = set(sample_text.split())
for k in sorted(vocabulary_dict):
...
elif k in samples:
...


Also, does it really matter if you iterate over vocabulary_dict in sorted order?

Not about performance but you should use with when dealing with files, so that closing is handled automatically:

with open('example.txt','w+') as f:
f.write(data)
# Closing is automatic


you can modify your list_textfiles function to use list comprehensions

def list_textfiles(directory):
return [directory + '/' + filename
for filename in listdir(directory) if filename.endswith(".cc")]
`

not much of an improvement but it's true that list comprehensions are faster.