3
\$\begingroup\$

This is my first project and I wrote it to practice coding. However, I'm wondering if there is an actual practical use for it. What this code does is list the number of times a word occurs in a text entry, either by alphabetical order, listing the frequency of the word or listing the index in the text entry where that word occurs. I am wondering what improvements or useful features could be added to it, and (as stated earlier) what is a real world application of this code.

from collections import Counter
from string import punctuation
import string
from typing import List
import operator


punctuate = ['\"',
             ',',
             '\"',
             "\'",
             "\'",
             '.' ,
             ":" ,
             ";" ,
             "!" ,
             "?" ,
             "\'",
             "\'",
             "'" ,
             "," ,
             "(" ,
             ")" ,
              "{",
             "}" ,
             "[" ,
             "]" ,
              '“',
              '”',
              "'",
              "’",
              '‘',
              '.',
              '#'




                 ] # text from websites use different quote marks than Pycharm so they where not registered
                   # ...but here are all the annoying punctuation marks
                   # that took forever to figure out how to decipher them in syntax and then get rid of them

print(punctuation) # makes a list of punctuation
print("Enter/Paste your multi-line input / paragraph of text. Hit Ctrl + D to run it ")
print("DO NOT PRESS ENTER OR CTRL+V AFTER THE FIRST TIME YOU PASTED IN AND RUN YOUR TEXT(UNLESS PASTED TEXT IS A SINGLE LINE WITH NO BREAKS ")
print("INSTEAD HIT CTRL + D TO RUN IT AFTER TEXT IS INPUTTED WHEN TERMINAL HAS STOPPED RUNNING AND THE INPUT '|' IS STILL AND FLASHING")

###$%^&*()(*&^%$%^&*()_)(*&^%$#$r%t^&*()_)(*&^%$%^&*()_)(*&^%$%^&*()_)(*&^%$#$%^&*()(*&^%$#$%^&*()(*&^%$#$%^&*()*&^%$#$%^&*()_+@#$%^&*()_#$%^&*()_

work_on_this_string_built_from_loop = r"" # this initializes out of the loop and adds
                                          # text from input as a single lined string

iteratory_count = 0 #initalized outside the loop to count the number of iteration required to process data..
                  # reports last iteration count accurately once CTRL + D is pushed

###$%^&*()(*&^%$%^&*()_)(*&^%$#$r%t^&*()_)(*&^%$%^&*()_)(*&^%$%^&*()_)(*&^%$#$%^&*()(*&^%$#$%^&*()(*&^%$#$%^&*()*&^%$#$%^&*()_+@#$%^&*()_#$%^&*()_




while True: # the loop will always = True so this creates an infinite loop
    iteratory_count += 1

    #print("DO NOT PRESS ENTER OR CTRL+V AFTER YOU HAVE INITIALLY RAN YOUR TEXT ") # this line will be used when trouble shooting output

    print()

    try:
        print(f"THE OUTPUT ABOVE IS NOT THE FINAL RESULT, THE PROGRAM IS NOT DONE YET: WE ARE NOW ON ITERATION {iteratory_count}")



        multi_lined_input = input("\n") # blank screen of input will appear after text is pasted in
        multi_lined_input=multi_lined_input.lower()
        multi_lined_input=multi_lined_input.replace('-', ' ') # this allows to get rid "-" while it is still a string, cannot do if string is in fact
                                                              # a string element that is part of an array of strings



        work_on_this_string_built_from_loop += multi_lined_input + ('' if multi_lined_input == ' ' else ' ') # this breaks lines officially
                                                                                                            # and stops the last character
                                                                                                            # of a line of input being deleted
                                                                                                            # because the else statement will repeat
                                                                                                            # the if statement within the loop


        #get_rid_of_refference_nums = list(range(0,10000000000) )
        raw_word_list = work_on_this_string_built_from_loop.split()

        get_rid_of_punctuation_marks = [item[:-1]  if item[-1] in punctuate  else item[1:]  if item[0] in punctuate else item for item in raw_word_list]
        #print(item[:-1]) # everything but last char
        #
        #print(item[-1]) # just the last char
        #
        #print(item[1:]) # everything but first char

        # Punctuation marks must be get ridden of because if the are at the end of word (like a '.'at the end of a sentence)
        # they will count the word with a trailing punctuation mark as a unique entry instead of being counted
        # with the rest of the words in the entry. ie,  'for!' wouldn't be counted with 'for'


        #first part of all_as_one makes item = every char in string element except last char IF
        # the last char of string element is an entry in puncutate (list of puntaution marks)
        #
        # second part of all_as_one makes item = every char in string element except  if the first char
        # is the first char of string element is an element within puncutate (list of puntaution)

        remove_consecutive_trailing_punctuation = [next_item[:-1] if next_item[-1] in punctuation  else next_item for next_item in get_rid_of_punctuation_marks]
        # same thing as the first part of all_as_one and removes if there is consecutive punctation marks at the end of the string element

        capitalized_array=[ capz.capitalize() for capz in remove_consecutive_trailing_punctuation]




        print()

        print()

        frequency_of_words_dict = dict(  Counter( capitalized_array )  )


        #print()
        #print(f"THIS LINE IS MADE TO TEST THE FINAL OUTPUT WHEN TROUBLE SHOOTING {frequency_of_words_dict}")

    except EOFError as error : # this allows for quiting the program using CTRL + D
                               # once the error is excepted, the program ends and this
                               #causes the break statement to execute and the loop ends
        print("ITERATING THROUGH EXCEPTION")


        print()
        keys_and_values_tuples_list = [(key, value) for key, value in frequency_of_words_dict.items()] # creates a list of tuples out of array
        #print(f"FOR TESTING TUPPLE LIST OUTPUT{keys_and_values_tuples_list}")
        word_max_key=max(keys_and_values_tuples_list)[0]
        max_val = max(keys_and_values_tuples_list)[1]


        print("This sorts all words within the file by alphabetical order and tells the number of occurrences ")
        for alphabetized_words, frequency_of_word in sorted(frequency_of_words_dict.items()): # sorted() sorts all the words alphabetically
                                                              # for loop REPORTS DATA AS  WORD : NUM_OCCURENCES

            assert isinstance(alphabetized_words, object) #just in case
            print()
            print(f"{alphabetized_words} : {frequency_of_word}")
            #print()
            #td= max(frequency_of_words_dict, key=lambda key: frequency_of_words_dict[key])
            #if j == td:

                #print(f" the word '{i}'occurs the most times with {j} occurences")
        break



print()
print("#########################################################################################################")
print("PROGRAM COMPLETE")

print(f"the length of words in entry is : {len(work_on_this_string_built_from_loop.split() )}")

print(f"but the total number of unique words in the entry is {len(frequency_of_words_dict)}")
print()
print("here is the list of every number and at what location it occurs")


for index, word in enumerate(capitalized_array) :  print(f"WORD: '{word}' INDEX : '{index}'")

most_frequent_words=[key for m in [max(frequency_of_words_dict.values())] for key, val in frequency_of_words_dict.items() if val == m]





print(f"The word(s) {most_frequent_words} occurs the most with { max(frequency_of_words_dict.values()) } occurrences.")


\$\endgroup\$
2
  • 2
    \$\begingroup\$ Welcome to CodeReview@SE. If you consider yourself a beginner in coding Python, please tag your question beginner. \$\endgroup\$
    – greybeard
    Mar 24, 2020 at 4:34
  • 2
    \$\begingroup\$ If this question is seriously about processing PDF, tag pdf. I did not see code to interpret PDF/ISO 32000-1 - contents is filtered / compressed more often than not. There are tools like PyPDF2. \$\endgroup\$
    – greybeard
    Mar 24, 2020 at 4:38

2 Answers 2

3
\$\begingroup\$

I think you need to work on naming things (I know, I know, naming things is hard). I would recommend to try to find slightly shorter names, and also names that do not contain the type of the variable. Usually it is enough to know if a variable contains one or multiple of something. It is fine to re-use a variable name if you just transform the element(s) it contains.

Therefore I would use the following names:

Current                                     Suggestion
#############################################################
punctuate                                   punctuation
iteratory_count                             i
multi_lined_input                           multiline_input
work_on_this_string_built_from_loop         user_input
raw_word_list                               words
get_rid_of_punctuation_marks                words
remove_consecutive_trailing_punctuation     words
capitalized_array                           words
frequency_of_words_dict                     word_frequencies
alphabetized_words                          word
frequency_of_word                           frequency

Names like remove_punctuation are not bad names. They are just not good names for variables. If you had a function named like that it would be perfectly fine and obvious what it does. And having a function like that might not be a bad idea as well.

In addition I would also like to encourage you to use whitespace sparingly (as recommended by PEP8) and to remove no longer needed code (at least before sharing your code with others).

Here's how your code would look with these suggestions:

# text from websites use different quote marks than Pycharm so they where not registered
# ...but here are all the annoying punctuation marks
# that took forever to figure out how to decipher them in syntax and then get rid of them

punctuation = ['\"', ',', '\"', "\'", '.', ":", ";", "!", "?" , "\'", "\'", "'" ,
               "(", ")", "{", "}" , "[" , "]" , '“', '”', "'", "’", '‘', '.', '#']
print(punctuation)
punctuation = "".join(punctuation)

print("Enter/Paste your multi-line input / paragraph of text. Hit Ctrl + D to run it ")
print("DO NOT PRESS ENTER OR CTRL+V AFTER THE FIRST TIME YOU PASTED IN AND RUN YOUR TEXT(UNLESS PASTED TEXT IS A SINGLE LINE WITH NO BREAKS ")
print("INSTEAD HIT CTRL + D TO RUN IT AFTER TEXT IS INPUTTED WHEN TERMINAL HAS STOPPED RUNNING AND THE INPUT '|' IS STILL AND FLASHING")

user_input = []
while True:
    print()
    try:
        multiline_input = input("\n").lower().replace('-', ' ').strip()
        if not multiline_input:
            continue
        user_input.append(multiline_input)
    except EOFError as error :
        print("FINISHED INPUT\n")
        break

user_input = " ".join(user_input)
words = [word.strip(punctuation).capitalize() for word in user_input.split()]
word_frequencies = dict(Counter(words))
word_max_key, max_val = max(word_frequencies.items())  # unused

print("This sorts all words within the file by alphabetical order and tells the number of occurrences ")
for word, frequency in sorted(word_frequencies.items()):
    print(f"\n{word} : {frequency}")

print("#########################################################################################################")
print("PROGRAM COMPLETE")

print(f"the length of words in entry is : {len(words)}")
print(f"but the total number of unique words in the entry is {len(word_frequencies)}")
print("\nhere is the list of every number and at what location it occurs")
for index, word in enumerate(words):
    print(f"WORD: '{word}' INDEX : '{index}'")

m = max(word_frequencies.values())
most_frequent_words = [key for key, val in word_frequencies.items() if val == m]
print(f"The word(s) {most_frequent_words} occurs the most with {m} occurrences.")

Note that I also moved the splitting into words out of the loop so that you only need to do it once and accumulated the user input in a list instead of doing repeated string addition, which is costly. Your whole cleanup code fits into one list comprehension by using str.strip (note that "abcba".strip("ab") == "c"). I also used tuple assignment instead of doing max(...) twice to get both the key and value of the maximum and cleaned up the code in the bottom.


If assert isinstance(alphabetized_words, object) ever fails, you will have done something horribly, horribly wrong. Everything (except for keywords) is an object in Python, so this has to be always true. It is true even for functions (also built-ins), classes (not only instances, but also the classes themselves, including built-ins) exceptions, basic types (like integers and lists) singletons (like None).

The only way I can think of for this to be false is if you unnecessarily implemented your own isinstance, shadowing the built-in, which has some bug.

\$\endgroup\$
4
  • \$\begingroup\$ In your naming suggestions, you name half the variables 'word'. While scalable, isn't that going to make debugging more complicated? \$\endgroup\$
    – Mast
    Mar 25, 2020 at 6:43
  • \$\begingroup\$ @Mast: Not really, IMO. 1. any error message will contain the linenumber, and the part to the right of the =. 2. If the code becomes too complicated for that (i.e. more than one line per transformation) it should be encapsulated in a function anyway, which makes it again obvious where the error is. \$\endgroup\$
    – Graipher
    Mar 25, 2020 at 7:15
  • \$\begingroup\$ @Mast: Also note what I added below the names, that some of the names the OP chose are not bad names, just that they would be better names for functions that do the transformation instead of the result of the transformation. \$\endgroup\$
    – Graipher
    Mar 25, 2020 at 7:16
  • \$\begingroup\$ Fair enough, thanks. \$\endgroup\$
    – Mast
    Mar 25, 2020 at 7:18
2
\$\begingroup\$

Comments and whitespace

Out of the 174 lines in your program, only 50 are lines of code (that list that's excessively spaced out is only counted as one). I would definitely think about what comments are needed in your program, i.e comments that explain why you did something should stay, and code that's just been commented out because it's not needed should be removed. Trim your whitespace too. While separating chunks of code can be helpful, you're overdoing it a little.

Operator Spacing

There should be a space before and after an operator in your program. Have a look:

name=input("Name: ") # WRONG
name = input("Name: ") # CORRECT

This is a convention outlined in PEP 8, the python convention guide.

Utilizing newlines

Instead of

print()
print("testing 123")

print("testing 123")
print()

you can do this

print("\ntesting 123")

print("testing 123\n")

Just a little quirk to reduce line count a bit, and reduces clutter since it doesn't require an additional print statement.

Chain function calls

Instead of

multi_lined_input = input("\n")
multi_lined_input=multi_lined_input.lower()
multi_lined_input=multi_lined_input.replace('-', ' ')

you should chain these calls:

multi_lined_input = input("\n").lower().replace('-', ' ')

Reduces line count and is still pretty clear what's going on.

Dict to list of tuples

There's a simpler way of converting a dict to a list of tuples. Casting the items of a dict to a list automatically does it for you. Have a look:

keys_and_values_tuples_list = list(frequency_of_words_dict.items())

Bug removing punctuation

When I enter "Linny", I expect to see Linny with a frequency of one. Instead I see "Linny with a frequency of one. This is because of the logic in your list comprehension. If it sees punctuation at the end, it will remove it and move onto the next word, instead of checking the beginning as well. The next line also only checks the end of the string. You can clean up this logic by utilizing some built in string methods, translate and maketrans. Have a look:

get_rid_of_punctuation_marks = [item.translate(str.maketrans('', '', string.punctuation)) for item in raw_word_list]

This removes all the punctuation by passing string.punctuation as a parameter to the translate function. This makes the next line with remove_consecutive_trailing_punctuation irrelevant. Then you can just use get_rid_of_punctuation_marks in your list comprehension when capitalizing the words.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.