Please tell me what the weak spots of this code are (specially in terms of efficiency) and how I can improve it:
def get_word_frequencies(filename):
handle = open(filename,'rU')
text = handle.read()
handle.close()
MUST_STRIP_PUNCTUATION = ['\n','&','-','"','\'',':',',','.','?','!'\
,';',')','(','[',']','{','}','*','#','@','~','`','\\','|','/','_'\
,'+','=','<','>','1','2','3','4','5','6','7','8','9','0']
text = text.lower()
for char in MUST_STRIP_PUNCTUATION:
if char in text:
text = text.replace(char,' ')
words_list = text.split(' ')
words_dict = {}
for word in words_list:
words_dict[word] = 0
for word in words_list:
words_dict[word] += 1
del words_dict['']
return words_dict
Some steps sound repetitive to me and it seems that I'm looping on the text many times but I think I'm obliged to take each of those steps separately(unless I'm wrong), for instance, replacing invalid characters should be on multiple separate iterations, or lower casing the whole text must be a separate step, and so on.
Also for creating the dictionary I'm suspicious a way better than words_dict[word] = 0
must exist?