2
\$\begingroup\$

I need to find a way how to create a frequency distribution out of multilple text files. In fact I am asked to check the number of times a word or a phrase occurs in a txt file. The code should check from a predefinded list (my list is called l1) how often this word or phrase can be found in the doc. My output should look like the following:

UNIQA VERSICHERUNGEN:31.12.2008
acceptance  2
acceptance credit   0
acceptance sampling 0
accounting principles   10
accounting principles board 0
additional  30
corporate   36
corporate bond  0
corporate finance   0
corporate governance    15

My code looks as follows. It does what it should do, but it is very slow (takes about a minute per file to process).

    from collections import Counter
    from itertools import chain
    import re
    import os
    import glob
    from nltk.tokenize import *
    import nltk
    from os import listdir

    def removeNonAscii(s): return "".join(i for i in s if ord(i)<128)

    def read_textfile(filename):
        # Reads the entire content of FILENAME and returns a non Ascii letters cleaned string
        infile = open(filename) 
        contents = removeNonAscii(infile.read())
        infile.close()
        return contents

    def list_textfiles(directory, min_file_size):
        # Creates a list of all files stored in DIRECTORY ending on '.txt'
        textfiles = []
    for root, dirs, files in os.walk(directory):
        for name in files:
            filename = os.path.join(root, name)
            if os.stat(filename).st_size > min_file_size:
                textfiles.append(filename)
    return textfiles

def remove_punctuation1(text):
    # Removes all punctuation and conotation from the string and returns a 'plain' string
    punctuation = '®©™€â´‚³©¥ã¼•ž®è±äüöž!@#“§$%^*()î_+€$=¿{”}[]:«;"»\â¢|<>,.?/~`0123456789'
    for sign in punctuation:
        text = text.replace(sign, "")
    return text

def remove_punctuation2(text):
    # Removes all punctuation and conotation from the string and returns a 'plain' string
    punctuation2 = '-&'
    for sign in punctuation2:
        text = text.replace(sign, " ")
    return text

filepath_dict = "H:/MA Daske/Wordlists/IFRS.txt" # input filepath for the used wordlist (here external accounting dictionary)
directory = "H:/Converted Text/EU0_OM0_FY2001" # directory of the text files to be processed
min_file_size = 90000

l1 = remove_punctuation2(removeNonAscii(read_textfile(filepath_dict))).lower().split('\n') # externally created word/expression list

vocabulary_dict  = {k:0 for k in l1} 

for filename in list_textfiles(directory, min_file_size):
    # inread each report as textfile, match tokenized text with predefined wordlist and count number of occurences of each element of that wordlist
    sample_text = remove_punctuation2(remove_punctuation1(read_textfile(filename).lower())).replace('\n', " ")
    #sample_text = remove_punctuation2(remove_punctuation1(sample_text)).replace('\n', " ")
    sample_text = ' '.join(sample_text.split())
    splitted = sample_text.split()
    c = Counter()
    c.update(splitted)
    #print(c)
    outfile = open(filename[:-4] + '_output' + '.txt', mode = 'w')
    string = str(filename)
    string_print = string[string.rfind('/')+1:string.find('-')] + ':' + string[-6:-4] + '.' + string[-8:-6] + '.' + string[-12:-8]
    samples = set(sample_text.split())
    for k in vocabulary_dict:
        spl = k.split()
        ln = len(spl)
        if ln > 1:
            check = re.findall(r'\b{0}\b'.format(k),sample_text)
            if check:
                vocabulary_dict[k] += len(check)
        elif k in samples:
            vocabulary_dict[k] += c[k]
    outfile.write(string_print + '\n')
    # line wise write each entry of the dictionary to the corresponding outputfile including comapany name, fiscal year end and tabulated frequency distribution
    for key, value in sorted( vocabulary_dict.items() ):
        outfile.write( str(key) + '\t' + str(value) + '\n' )
    outfile.close()
\$\endgroup\$
  • 1
    \$\begingroup\$ Welcome to CodeReview.SE! Indentation seems to be wrong, could you please have a look and fix ? \$\endgroup\$ – SylvainD Mar 1 '15 at 13:12
2
\$\begingroup\$

Use consistent naming

removeNonAscii(s) -> remove_non_ascii(n)

Use with

def read_textfile(filename):
    # Reads the entire content of FILENAME and returns a non Ascii letters cleaned string
    with open(filename) as f:
        return removeNonAscii(f.read())

Merge similar functions

def remove_punctuation1(text):
    # Removes all punctuation and conotation from the string and returns a 'plain' string
    punctuation = '®©™€â´‚³©¥ã¼•ž®è±äüöž!@#“§$%^*()î_+€$=¿{”}[]:«;"»\â¢|<>,.?/~`0123456789'
    for sign in punctuation:
        text = text.replace(sign, "")
    return text

    def remove_punctuation(text):
    # Removes all punctuation and conotation from the string and returns a 'plain' string
    punctuation2 = '-&'
    for sign in punctuation2:
        text = text.replace(sign, " ")
    return text

Should become

def remove_punctuation(text):
    # Removes all punctuation and conotation from the string and returns a 'plain' string
    punctuation2 = '-&'+'®©™€â´‚³©¥ã¼•ž®è±äüöž!@#“§$%^*()î_+€$=¿{”}[]:«;"»\â¢|<>,.?/~`0123456789'
    for sign in punctuation2:
        text = text.replace(sign, " ")
    return text

Use docstrings

def read_textfile(filename):
    """
    Reads the entire content of FILENAME and 
    returns a non Ascii letters cleaned string.
    """
    with open(filename) as f:
        return removeNonAscii(f.read())

Avoid one-two letter variable names, prefer longer ones

l1 -> words_to_check
\$\endgroup\$
2
\$\begingroup\$
  • It would make a lot more sense to read files one line at a time (less strain on the runtime, allocating only a small string at a time might be a lot easier / efficient than trying to work with a whole file at once).

  • It's better to leave out particularities such as input file location. Instead read them in from command line (with possibly some sensible defaults).

  • Your remove_punctuation2 has two problems: numbers in function names are considered bad practice (how is this function different from remove_punctuation1? why not reflect this difference in the name?). Another problem is that what it does could be done with a lot less effort with a regular expression. Look at re.sub for details.

  • Why go into so much trouble creating string_print? It seems like it could be less opaque to the reader what the result should look like if you used a template or named the pieces of strings you combined to produce this string.

  • I don't see where are you use NLTK library? Did you need to import it?

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.