# Program to count common words in documents

I finished my first program in Ruby and I would like to share it with you so I can get some suggestions or recommendations. I'd really like to hear them because I am learning and I want to have a solid base. The program has a basic function: it counts the 15 more common words in a document. It is useful in literary studies because it allows the user to know which words is repeating more times, maybe without noticing it. This is how the program works:

The user is able to drag the document into the terminal and to explicitly write the path. Then, the script uses the gem 'yomu' to read it. It reads pdf and docx as well as other files that aren't important for me now. After that, the script makes an array of the words of the document. At this point, the script removes words like prepositions or connectors which aren't important to the user. Finally, it outputs the results in a table through a gem called 'hirb'.

The scripts also offers the possibility to modify the list of words that are going to be removed from the original words. For example, if a document has "Written by Rick Deckard" in every page as a footnote, the user is able to include the words "Written", "by", "Rick" and "Deckard" in the list and they aren't going to be counted. The user can modify the list and save the changes for that session or for ever.

So, there are three scripts: "wordsworth.rb", "core.rb", "forbidden_list_modify_eng.rb". The former script is in the main directory, while the other two are in the path 'languages/english'. That's because I adapted the program to spanish and catalan as well. Anyway, the main script is "wordsworth.rb", which asks the user their language and redirect him or her to the chosen language folder (english, in this case). Once in the english section, the script calls "core.rb", which is the main script of the english section. It means that this scripts is going to do the chief task of the program: count the words and output the results. However, "core.rb" offers the user to modify the list, and this option redirects the user to the third script "forbidden_list_modify_eng.rb" (sorry for the name).

Now I am going to post the three scripts and explain them in detail:

#needs to install 'yomu' and 'hirb' gems

require 'hirb'
require 'yomu'
require 'fileutils'

require_relative "languages/english/forbidden_list_modify_eng.rb"
require_relative "languages/english/core.rb"

def language
puts "1. Català"
puts "2. English"
puts "3. Castellano"
idiom = gets.chomp.to_s
case idiom
when "1", "català", "Català", "CATALÀ"
system "clear" or system "cls"
when "2", "english", "English", "ENGLISH"
system "clear" or system "cls"
when "3", "castellano", "Castellano", "CASTELLANO"
system "clear" or system "cls"
else
system "clear" or system "cls"
puts "You have to choose a language in the list."
puts "-----------------------------------------------------"
language
end
end

language()


This script is pretty straight-forward, except for the last part, which I'll explain later. It first requires the needed gems and calls the needed scripts. After that, a menu asks the language to the user so it can redirect him or her to the correct language. We suppose the user choose 2 and gets redirected to the function "main_menu", which lands the user to the "core.rb" script:

def tutorial
puts "The program works like that... (blah blah blah)"
end

english_list = retrieve_forbidden_words.split("\n")
puts "Choose what you want to do by writing its number:"
puts "1: Analyze a document."
puts "2: Modify the list of forbidden words."
puts "3: Help me to use this program."
puts "4: Exit the program."
main_choice = gets.chomp.to_s
case main_choice
when "1"
system "clear" or system "cls"
english(english_list)
when "2"
system "clear" or system "cls"
when "3"
system "clear" or system "cls"
tutorial
when "4"
exit
else
end
end

puts "Drag in here the document you want to analyze:"
provided_path = gets.chomp.to_s.tr("'", "").rstrip
end

def execution (provided_path, forbidden_list)

text_listed = doc.to_s.downcase.gsub(/\p{^Alnum}/, " ").split(" ")
pruned_text = text_listed - forbidden_list

frequency = Hash.new 0
pruned_text.each { |word| frequency[word] += 1 }
sorted_frequency = Hash[frequency.sort_by{ | word, times | -times }[0..20]]
toptimes = sorted_frequency.values
topwords = sorted_frequency.keys

puts Hirb::Helpers::AutoTable.render(sorted_frequency, headers: {0 => 'Word', 1 => 'Times'}, description: false)
puts "-----------------------------------------------------"

end


In the "main_menu" the user can choose to analyze the document. If so, the script asks the path of the document in the function "user_doc_path", which takes also the list of forbidden words in English from a .txt file which comes in the program by default. It is a simple file named "english_list.txt" which contains all the words to be removed from the document separated by a line break. So, the user provides the path of the document and the script opens it using 'yum' gem and sort it to get the words in an array. After that, the script creates a hash that stores the number of times each word is repeated throughout the text. Then, the program orders the hash, reverses it and get the 15 top results which finally output in a table provided by 'hirb' gem.

If the user choose to modify the list of words that are removed from the document, he or she is redirected to the "forbidden_list_eng_modify.rb":

def remove_words(list_to_remove)
puts "Write the words you want to remove from the list. If there are more than one, divide them by commas."
remove_propose = gets.chomp.tr(" ", "").to_s.split(",")
new_list = list_to_remove - remove_propose

File.new("languages/english/english_backup.txt", "w")
File.open("languages/english/english_backup.txt", "w+") do |f|
f.puts(list_to_remove)
end

File.open("languages/english/english_list.txt", "w+") do |f|
f.puts(new_list)
end

puts "The words have been removed. Do you want to see the new list? [Yes/No]"
showit = gets.chomp.tr(" ", "").to_s
case showit
when "Yes", "yes", "Y", "y"
system "clear" or system "cls"
print new_list
puts "-----------------------------------------------------"
when "No", "no", "N", "n"
system "clear" or system "cls"
end
end

puts "Write the words you want to add to the list. If there are more than one, divide them by commas."

File.new("languages/english/english_backup.txt", "w")
File.open("languages/english/english_backup.txt", "w+") do |f|
end
File.open("languages/english/english_list.txt", "w+") do |f|
f.puts(new_list)
end

puts "The words have been added. Do you want to see the new list? [Yes/No]"
showit = gets.chomp.tr(" ", "").to_s
case showit
when "Yes", "yes", "Y", "y"
system "clear" or system "cls"
print new_list
puts "-----------------------------------------------------"
when "No", "no", "N", "n"
puts "-----------------------------------------------------"
end
end

original_list = retrieve_forbidden_words.split("\n")

puts "This list contains all the words that are automatically removed from the document you want to analyze. The words in it are widely-used words with no special meaning like connectors or prepositions. However, you can modify the list as you want."
puts "1: Show all the words of the list."
puts "2: Remove words."
puts "4: Save the new list for other sessions."
list_choice = gets.chomp.to_s
case list_choice
when "1"
system "clear" or system "cls"
print original_list
puts ""
puts "-----------------------------------------------------"
when "2"
system "clear" or system "cls"
remove_words(original_list)
when "3"
system "clear" or system "cls"
when "4"
backup_existence = File.file?('languages/english/english_backup.txt')
if backup_existence == TRUE
File.delete('languages/english/english_backup.txt')
system "clear" or system "cls"
puts "Changes has been succesfully saved."
puts "-----------------------------------------------------"
end
if backup_existence == FALSE
system "clear" or system "cls"
puts "The list hasn't been modified, so it's nothing to save."
puts "-----------------------------------------------------"
end
when "5"
system "clear" or system "cls"
else
system "clear" or system "cls"
end
end


This script shows the user a menu which offers the possibility of showing the list, adding or removing some words and saving the changes for ever. The first option, show the complete list, just prints the output of the "english_list.txt", which is the file where the words are stored. The "add_words", as the "remove_words" function, asks the user to write the words that want to get removed separating them by commas. The script just removes the provided array from the default list. At this point, the script creates a "english_backup.txt" which is the back up in case the user closes the program without saving. The modified list is named "english_list.txt", and it works as the new default list, so the user can analyze more texts in the same session with that list. If the user chooses to save the changes, the "english_backup.txt" is destroyed and the new "english_list.txt" is the main list. If the user exits, the "english_backup.txt" will get its former name ("english_list.txt") and the unsaved list file will be removed. This is achieved in the former script, "wordsworth.rb", when END is detected it proceeds to remove the unsaved file and restore the backup as main list.

• Just wanted to let you know that I really like the question, and I've been working on a review. I'll post it (and delete this comment) at some point in the next few days. – thesecretmaster Feb 21 '18 at 7:27
• Hello! That makes me really happy! I thought the question was forgotten in the stackexhange unstoppable flow. I'm looking forward to reading your review :) – H4ml3tt3d Feb 21 '18 at 10:25

This is a really interesting project, and looks pretty solid. I'm just going to go through and give you some ruby style tips as I see them. I've annotated all of the files with my thoughts.

First of all, you use lists in a couple places. Because of that, I'd suggest that you create a generic list method right above your #language method:

def list(*options, retry_msg: '')
options = Array(options).flatten.map(&:to_s).uniq
invalid_opts = (0..options.length).map(&:to_s)
throw "You cannot pass a number as an option" if options.any? { |opt| invalid_opts.include?(opt) }
loop do
puts options.map.with_index { |opt, i| "#{i}. #{opt}" }
selection = gets.chomp
system "clear" or system "cls"
if (0..options.length-1).map(&:to_s).include? selection
return options[selection.to_i]
else
selection = options.select { |opt| opt.downcase == selection.downcase }.first
return selection unless selection.nil?
end
puts retry_msg
end
end


Side note: Since this would remove your lists throughout, I just wanted to add some code style tips in case you decided to leave the list in. Here are my comments on the list in your #language method:

def language
# I would simplify the listing of languages (and make it more extensible)
# by having an array of languages you support.
# I've created the array using the %w literal, which creates an array
# of words split on the spaces.
languages = %w[Català English Castellano]
# puts "1. Català"
# puts "2. English"
# puts "3. Castellano"
# And then print them out: (puts automatically adds newlines
# between the array elements)
puts languages.map.with_index { |lang, index| "#{index}. #{lang}" }
idiom = gets.chomp.to_s
# Here, you can convert the idiom to lowercase so you don't need
# to have different capitalizations in your when statements
case idiom.downcase
when "1", "català"#, "Català", "CATALÀ"
system "clear" or system "cls"
when "2", "english"#, "English", "ENGLISH"
system "clear" or system "cls"
when "3", "castellano"#, "Castellano", "CASTELLANO"
system "clear" or system "cls"
else
system "clear" or system "cls"
puts "You have to choose a language in the list."
puts '-'*53 # You can use multiplication instead of typing the char so many times
language
end
end


Anyways, here's an actual review of your main file:

require 'hirb'
require 'yomu'
require 'fileutils'

require_relative "languages/english/forbidden_list_modify_eng.rb"
require_relative "languages/english/core.rb"

# The addition of the list method means that all this code doesn't need
# to be wrapped in a language method
# def language
# puts "1. Català"
# puts "2. English"
# puts "3. Castellano"
# idiom = gets.chomp.to_s
retry_msg  = "You have to choose a language in the list.\n"
retry_msg += '-'*53
case list(%w[Català English Castellano], retry_msg: retry_msg)
when "Català"
when "English"
when "Castellano"
end
# end

# language()


And, for the second file:

def tutorial
# If this is really long, it might be a good idea to put
# the contents of the tutorial in a text file and instead
puts "The program works like that... (blah blah blah)"
end

# I've renamed the variable to be forbidden_words (from retrieve_forbidden_words) because the variable is the forbidden words, not the action of retrieving them.
english_list = forbidden_words.split("\n")
# Here we can use the magic list method
puts "Choose what you want to do by writing its number:"
# puts "1: Analyze a document."
# puts "2: Modify the list of forbidden words."
# puts "3: Help me to use this program."
# puts "4: Exit the program."
# main_choice = gets.chomp.to_s
# case main_choice
options = []
options << 'Analyze a document.'
options << 'Modify the list of forbidden words.'
options << 'Help me to use this program.'
options << 'Exit the program.'
case list(options)
when 'Analyze a document.'
english(english_list)
when 'Modify the list of forbidden words.'
when 'Help me to use this program.'
tutorial
when 'Exit the program.'
exit
end
end

puts "Drag in here the document you want to analyze:"
provided_path = gets.chomp.to_s.tr("'", "").rstrip
end

def execution (provided_path, forbidden_list)

text_listed = doc.to_s.downcase.gsub(/\p{^Alnum}/, " ").split(" ")
pruned_text = text_listed - forbidden_list

frequency = Hash.new 0
pruned_text.each { |word| frequency[word] += 1 }
sorted_frequency = Hash[frequency.sort_by{ | word, times | -times }[0..20]]
toptimes = sorted_frequency.values
topwords = sorted_frequency.keys

puts Hirb::Helpers::AutoTable.render(sorted_frequency, headers: {0 => 'Word', 1 => 'Times'}, description: false)
puts "-----------------------------------------------------"

end


And, the final file:

def remove_words(list_to_remove)
puts "Write the words you want to remove from the list. If there are more than one, divide them by commas."
remove_propose = gets.chomp.tr(" ", "").to_s.split(",")
new_list = list_to_remove - remove_propose

File.new("languages/english/english_backup.txt", "w")
File.open("languages/english/english_backup.txt", "w+") do |f|
f.puts(list_to_remove)
end

File.open("languages/english/english_list.txt", "w+") do |f|
f.puts(new_list)
end

puts "The words have been removed. Do you want to see the new list? [Yes/No]"
showit = gets.chomp.tr(" ", "").to_s
case showit
when "Yes", "yes", "Y", "y"
system "clear" or system "cls"
print new_list
puts "-----------------------------------------------------"
when "No", "no", "N", "n"
system "clear" or system "cls"
end
end

puts "Write the words you want to add to the list. If there are more than one, divide them by commas."

File.new("languages/english/english_backup.txt", "w")
File.open("languages/english/english_backup.txt", "w+") do |f|
end
File.open("languages/english/english_list.txt", "w+") do |f|
f.puts(new_list)
end

puts "The words have been added. Do you want to see the new list? [Yes/No]"
showit = gets.chomp.tr(" ", "").to_s
case showit
when "Yes", "yes", "Y", "y"
system "clear" or system "cls"
print new_list
puts "-----------------------------------------------------"
when "No", "no", "N", "n"
puts "-----------------------------------------------------"
end
end

original_list = retrieve_forbidden_words.split("\n")

puts "This list contains all the words that are automatically removed from the document you want to analyze. The words in it are widely-used words with no special meaning like connectors or prepositions. However, you can modify the list as you want."
puts "1: Show all the words of the list."
puts "2: Remove words."
puts "4: Save the new list for other sessions."
list_choice = gets.chomp.to_s
case list_choice
when "1"
system "clear" or system "cls"
print original_list
puts ""
puts "-----------------------------------------------------"
when "2"
system "clear" or system "cls"
remove_words(original_list)
when "3"
system "clear" or system "cls"
when "4"
backup_existence = File.file?('languages/english/english_backup.txt')
if backup_existence == TRUE
File.delete('languages/english/english_backup.txt')
system "clear" or system "cls"
puts "Changes has been succesfully saved."
puts "-----------------------------------------------------"
end
if backup_existence == FALSE
system "clear" or system "cls"
puts "The list hasn't been modified, so it's nothing to save."
puts "-----------------------------------------------------"