# Frequency distribution of words and phrases

I need to find a way how to create a frequency distribution out of multilple text files. In fact I am asked to check the number of times a word or a phrase occurs in a txt file. The code should check from a predefinded list (my list is called l1) how often this word or phrase can be found in the doc. My output should look like the following:

UNIQA VERSICHERUNGEN:31.12.2008
acceptance  2
acceptance credit   0
acceptance sampling 0
accounting principles   10
accounting principles board 0
corporate   36
corporate bond  0
corporate finance   0
corporate governance    15


My code looks as follows. It does what it should do, but it is very slow (takes about a minute per file to process).

    from collections import Counter
from itertools import chain
import re
import os
import glob
from nltk.tokenize import *
import nltk
from os import listdir

def removeNonAscii(s): return "".join(i for i in s if ord(i)<128)

# Reads the entire content of FILENAME and returns a non Ascii letters cleaned string
infile = open(filename)
infile.close()
return contents

def list_textfiles(directory, min_file_size):
# Creates a list of all files stored in DIRECTORY ending on '.txt'
textfiles = []
for root, dirs, files in os.walk(directory):
for name in files:
filename = os.path.join(root, name)
if os.stat(filename).st_size > min_file_size:
textfiles.append(filename)
return textfiles

def remove_punctuation1(text):
# Removes all punctuation and conotation from the string and returns a 'plain' string
punctuation = '®©™€â´‚³©¥ã¼•ž®è±äüöž!@#Â“§$%^*()î_+€$=¿{”}[]:«;"»\â¢|<>,.?/~0123456789'
text = text.replace(sign, "")
return text

def remove_punctuation2(text):
# Removes all punctuation and conotation from the string and returns a 'plain' string
punctuation2 = '-&'
text = text.replace(sign, " ")
return text

filepath_dict = "H:/MA Daske/Wordlists/IFRS.txt" # input filepath for the used wordlist (here external accounting dictionary)
directory = "H:/Converted Text/EU0_OM0_FY2001" # directory of the text files to be processed
min_file_size = 90000

l1 = remove_punctuation2(removeNonAscii(read_textfile(filepath_dict))).lower().split('\n') # externally created word/expression list

vocabulary_dict  = {k:0 for k in l1}

for filename in list_textfiles(directory, min_file_size):
# inread each report as textfile, match tokenized text with predefined wordlist and count number of occurences of each element of that wordlist
#sample_text = remove_punctuation2(remove_punctuation1(sample_text)).replace('\n', " ")
sample_text = ' '.join(sample_text.split())
splitted = sample_text.split()
c = Counter()
c.update(splitted)
#print(c)
outfile = open(filename[:-4] + '_output' + '.txt', mode = 'w')
string = str(filename)
string_print = string[string.rfind('/')+1:string.find('-')] + ':' + string[-6:-4] + '.' + string[-8:-6] + '.' + string[-12:-8]
samples = set(sample_text.split())
for k in vocabulary_dict:
spl = k.split()
ln = len(spl)
if ln > 1:
check = re.findall(r'\b{0}\b'.format(k),sample_text)
if check:
vocabulary_dict[k] += len(check)
elif k in samples:
vocabulary_dict[k] += c[k]
outfile.write(string_print + '\n')
# line wise write each entry of the dictionary to the corresponding outputfile including comapany name, fiscal year end and tabulated frequency distribution
for key, value in sorted( vocabulary_dict.items() ):
outfile.write( str(key) + '\t' + str(value) + '\n' )
outfile.close()

• Welcome to CodeReview.SE! Indentation seems to be wrong, could you please have a look and fix ? – SylvainD Mar 1 '15 at 13:12

## Use consistent naming

removeNonAscii(s) -> remove_non_ascii(n)


## Use with

def read_textfile(filename):
# Reads the entire content of FILENAME and returns a non Ascii letters cleaned string
with open(filename) as f:


## Merge similar functions

def remove_punctuation1(text):
# Removes all punctuation and conotation from the string and returns a 'plain' string
punctuation = '®©™€â´‚³©¥ã¼•ž®è±äüöž!@#Â“§$%^*()î_+€$=¿{”}[]:«;"»\â¢|<>,.?/~0123456789'
text = text.replace(sign, "")
return text

def remove_punctuation(text):
# Removes all punctuation and conotation from the string and returns a 'plain' string
punctuation2 = '-&'
text = text.replace(sign, " ")
return text


Should become

def remove_punctuation(text):
# Removes all punctuation and conotation from the string and returns a 'plain' string
punctuation2 = '-&'+'®©™€â´‚³©¥ã¼•ž®è±äüöž!@#Â“§$%^*()î_+€$=¿{”}[]:«;"»\â¢|<>,.?/~0123456789'
text = text.replace(sign, " ")
return text


## Use docstrings

def read_textfile(filename):
"""
Reads the entire content of FILENAME and
returns a non Ascii letters cleaned string.
"""
with open(filename) as f:


## Avoid one-two letter variable names, prefer longer ones

l1 -> words_to_check

• It would make a lot more sense to read files one line at a time (less strain on the runtime, allocating only a small string at a time might be a lot easier / efficient than trying to work with a whole file at once).

• It's better to leave out particularities such as input file location. Instead read them in from command line (with possibly some sensible defaults).

• Your remove_punctuation2 has two problems: numbers in function names are considered bad practice (how is this function different from remove_punctuation1? why not reflect this difference in the name?). Another problem is that what it does could be done with a lot less effort with a regular expression. Look at re.sub for details.

• Why go into so much trouble creating string_print`? It seems like it could be less opaque to the reader what the result should look like if you used a template or named the pieces of strings you combined to produce this string.

• I don't see where are you use NLTK library? Did you need to import it?