8
\$\begingroup\$

I am working on this Python project, and I pretty much have it done, I just wanted to get a second opinion, to make sure it looks good, and to see if anyone had any tips for it. I also was a little confused by the instructions I was given by my instructor for the project, so I want to make sure the program I wrote actually does what the instruction ask it to do. Any feedback would be much appreciated. Thanks!

Here are the instruction I was given:

A file concordance tracks the unique words in a file and their frequencies. Write a program that displays a concordance for a file. The program should output the unique words and their frequencies in alphabetical order. Variations are to track sequences of two words and their frequencies, or n words and their frequencies.

Here is my program:

#read the file from the user

file_name=input("Enter the file name which is in the computer : ")

#open the read file with read mode

file = open(file_name,"r")

#initialise the empty dictionary
 
words_freq={}

#read file and split the words 
#and iterate over each word in the file
#if new word is recognised it will be added to word_freq dictionary
#else increment the word count

for w in file.read().split():

        if w not in words_freq:

                words_freq[w] = 1

        else:

                words_freq[w] += 1

#closing the input file

file.close();

#sort the list and print the word the its frequency

for k in sorted(words_freq):
 
        print(k,":",words_freq[k])
\$\endgroup\$
3
  • 1
    \$\begingroup\$ Welcome to Code Review! Have you tested the code to see if it works as expected? \$\endgroup\$ Feb 18, 2022 at 19:24
  • \$\begingroup\$ Hi @SᴀᴍOnᴇᴌᴀ the code does run, I am just not completely sure if it does what the instructions ask it to do. I found them kind of confusing, so I just tried my best based on what I thought they wanted. \$\endgroup\$ Feb 18, 2022 at 19:49
  • \$\begingroup\$ I've updated the title of your post to make it in line with this site's requirement. The title should describe what your program does, not what you are looking for. \$\endgroup\$
    – AJNeufeld
    Feb 18, 2022 at 20:57

2 Answers 2

10
\$\begingroup\$

Your current program is reasonable, but here are a few suggestions. (1) Most important of all, start putting your code in functions. This is one of the keys to successful programming. (2) Under most circumstances, open files via a context manager. (3) When counting things, use a Counter: it's like a dict, but easier to use because you don't need to worry about the existence of keys and because you can update the counter by feeding it an iterable of things. (4) Prefer command-line arguments (sys.argv) over interactivity (input). (5) Be precise when referring to elements of file paths (path vs name). (6) If your program might need to process extremely large files, process them line-by-line rather than slurping them in one gulp.

import sys
from collections import Counter

def main(args):
    # In this function we handle user input and output.
    # All other work is delegated to data-oriented functions.
    file_path = args[0] if args else input("Enter file path: ")
    freq = get_word_frequencies(file_path)
    for word, n in sorted(freq.items()):
        print(word, n)

def get_word_frequencies(file_path):
    # Takes a file path. Returns a frequency tally of words.
    freq = Counter()
    with open(file_path) as file:     # Use a context manager.
        for line in file:             # Line by line processing.
            words = line.split()
            freq.update(words)        # Counters are handy.
    return freq

if __name__ == '__main__':
    main(sys.argv[1:])

To emphasize the ease of using a Counter, we could also implement the function like this, provided that we were not worried about dealing with very large files.

def get_word_frequencies(file_path):
    with open(file_path) as file:
        return Counter(file.read().split())
\$\endgroup\$
2
\$\begingroup\$

PEP 8

The Style Guide for Python Code lists many guidelines that all Python programmers should follow. Among the many guidelines, it recommends a space around binary operators, and a space after commas.

file_name=input(…) should be file_name = input(…)

words_freq={} should be words_freq = {}

print(k,":",words_freq[k]) should be print(k, ":", words_freq[k])

Unique Words

If your input file contained

It was the best of times; it was the worst of times.

How many occurrences of “it” will your program output? How many should it output? Why are these not the same?

Because the input is in mixed case, your output will have one occurrence of “it” and one occurrence of “It”. You can fix this by converting all words to either lowercase or uppercase. An even better option is to use casefold() on the input, to convert to lowercase as well as convert equivalent letter representations to a common format.

line = line.casefold()

Valid words

How many occurrences of “times” is there? Zero! How many did you expect? Two? What’s wrong?

Your split only splits the input on white-space. This means you get words like ”times;” and “times.” but no ”times” exists.

You can leverage the regular expression engine to extract strings which are only composed of “word” characters:

import re

…

for line in file:
    line = line.casefold()
    words = re.findall(r"\w+", line)
    for word in words:
        …
…

Now, words will contain two occurrences of "times" and no semicolon or period characters.

… except some words do have punctuation characters, such as apostrophes. If your input contained can’t, the above would split that into two words: "can" and "t"! We’d have to improve our regular expression to not only accept word characters with \w, but also the apostrophe:

words = re.findall(r"[\w’]+", line)

Note “Word characters” additionally include numbers and underscores, so you may want to study the regular expression syntax to see how you can include only letters in the resulting words.

Double-word concordance

In the above input, the pairs of words “it was”, “was the”, and “of times” all occur twice. Let’s automate the counting of that. Assuming Python 3.10 or later:

import re
from collections import Counter
from itertools import pairwise

line = "It was the best of times; it was the worse of times."

line = line.casefold()
words = re.findall(r"[\w+‘]+", line)

freq = Counter(pairwise(words))
print(freq)

pairwise() takes the items of words and returns the first and second as a tuple, then it returns the second and third as a tuple, then it returns the third and fourth as a tuple, and so on. I’m leveraging the Counter class described by FMc to count occurrences of these tuples of two words.

The itertools package also shows you a triplewise() recipe which could be used to count tuples of three words.

Except …

Generators

It was the best of times, it was the worst of times, it was the
age of wisdom, it was the age of foolishness, it was the epoch of
belief, it was the epoch of incredulity, it was the season of
light, it was the season of darkness, it was the spring of hope,
it was the winter of despair, …

If your file contains multiple lines, reading the file line by line, and processing each line looking for word pairs is going to ignore the pair with the “the” at the end of the first line and the “age” at the beginning of the second line. We need to remember the last word (or words) from the previous line when we start processing the next line if we want 2-word (n-word) concordance of the entire file.

You can revert to your original method of reading the file into one long string. You’d casefold that string, making a second copy of the entire file in memory, and then find all words in that long string, effectively making a third copy of a potentially huge file. While it works, there is a more memory-friendly approach.

When you open a file, you get a “file handle” which can be used as a generator that returns all the lines of the file, one at a time. This is what happens when you write:

with open(file_name) as file:
    for line in file:
        …

The for statement asks file for a line, which it uses in the body of the statement, then it asks for the next line, and then the next line. The entire file is not read into memory as one monolithic block, rather only as much of the file as required is read such that a complete “line” can be returned.

We can rewrite the above code to make the generator explicit.

with open(file_name) as file:
    line_generator = (line for line in file)
    for line in line_generator:
        …

This new generator isn’t adding any value to the processing. Let’s make it actually do something interesting:

with open(file_name) as file:
    lowercase_line_generator = (line.casefold() for line in file)
    for line in lowercase_line_generator:
        …

Now, when for line in lowercase_line_generator requests a new line, the generator will request a line from file, and casefold() it into the lowercase equivalent and return that.

We’re using one generator as the input to the next generator. Let’s extend that with yet another generator.

with open(file_name) as file:
    lowercase_line_generator = (line.casefold() for line in file)
    word_generator = (word for line in lowercase_line_generator
                           for word in re.finditer(r"[\w’]+", line)
    for word in word_generator:
        …

Our word generator returns one word at a time. To get the words, it asks the line generator for a line and passes the line to finditer, which looks for and returns the individual words one at a time. When it runs out of words in that line, the line generator will retrieve the next line from the line generator, and pass it to finditer. The result: all the words in the file returned one at a time, without being interrupted by end of lines, without reading the entire file into memory.

We can pass that generator to pairwise() and count each successive word pair:

with open(file_name) as file:
    lowercase_line_generator = (line.casefold() for line in file)
    word_generator = (word for line in lowercase_line_generator
                           for word in re.finditer(r"[\w’]+", line)
    pair_generator = pairwise(word_generator)
    freq = Counter(pair_generator)

This could actually be done in one statement …

with open(file_name) as file:
    freq = Counter(pairwise(word for line in file
                                 for word in re.finditer(r"[\w’]+",
                                                         line.casefold()))

… but there can be an advantage to leaving generators as their own entities. You can conditionally combine them in different ways to achieve a variety of effects:

with open(file_name) as file:
    words = (word for line in file
                  for word in re.finditer(r"[\w’]+", line.casefold()))

    if track_length == 1:
        track = words
    elif track_length == 2:
        track = pairwise(words)
    elif track_length == 3:
        track = triplewise(words)

    freq = Counter(track)
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.