Program to count common words in documents

Question

I finished my first program in Ruby and I would like to share it with you so I can get some suggestions or recommendations. I'd really like to hear them because I am learning and I want to have a solid base. The program has a basic function: it counts the 15 more common words in a document. It is useful in literary studies because it allows the user to know which words is repeating more times, maybe without noticing it. This is how the program works:

The user is able to drag the document into the terminal and to explicitly write the path. Then, the script uses the gem 'yomu' to read it. It reads pdf and docx as well as other files that aren't important for me now. After that, the script makes an array of the words of the document. At this point, the script removes words like prepositions or connectors which aren't important to the user. Finally, it outputs the results in a table through a gem called 'hirb'.

The scripts also offers the possibility to modify the list of words that are going to be removed from the original words. For example, if a document has "Written by Rick Deckard" in every page as a footnote, the user is able to include the words "Written", "by", "Rick" and "Deckard" in the list and they aren't going to be counted. The user can modify the list and save the changes for that session or for ever.

So, there are three scripts: "wordsworth.rb", "core.rb", "forbidden_list_modify_eng.rb". The former script is in the main directory, while the other two are in the path 'languages/english'. That's because I adapted the program to spanish and catalan as well. Anyway, the main script is "wordsworth.rb", which asks the user their language and redirect him or her to the chosen language folder (english, in this case). Once in the english section, the script calls "core.rb", which is the main script of the english section. It means that this scripts is going to do the chief task of the program: count the words and output the results. However, "core.rb" offers the user to modify the list, and this option redirects the user to the third script "forbidden_list_modify_eng.rb" (sorry for the name).

Now I am going to post the three scripts and explain them in detail:

#needs to install 'yomu' and 'hirb' gems

require 'hirb'
require 'yomu'
require 'fileutils'

require_relative "languages/english/forbidden_list_modify_eng.rb"
require_relative "languages/english/core.rb"

def language
  puts "Choose your language:"
  puts "1. Català"
  puts "2. English"
  puts "3. Castellano"
  idiom = gets.chomp.to_s
  case idiom
  when "1", "català", "Català", "CATALÀ"
    system "clear" or system "cls"
    menu_principal_cat
  when "2", "english", "English", "ENGLISH"
    system "clear" or system "cls"
    main_menu
  when "3", "castellano", "Castellano", "CASTELLANO"
    system "clear" or system "cls"
    menu_principal_cast
  else
    system "clear" or system "cls"
    puts "You have to choose a language in the list."
    puts "-----------------------------------------------------"
    language
  end
end

language()

This script is pretty straight-forward, except for the last part, which I'll explain later. It first requires the needed gems and calls the needed scripts. After that, a menu asks the language to the user so it can redirect him or her to the correct language. We suppose the user choose 2 and gets redirected to the function "main_menu", which lands the user to the "core.rb" script:

def tutorial
  puts "The program works like that... (blah blah blah)"
end

def main_menu
  retrieve_forbidden_words = File.read('languages/english/english_list.txt')
  english_list = retrieve_forbidden_words.split("\n")
  puts "Choose what you want to do by writing its number:"
  puts "1: Analyze a document."
  puts "2: Modify the list of forbidden words."
  puts "3: Help me to use this program."
  puts "4: Exit the program."
  main_choice = gets.chomp.to_s
  case main_choice
  when "1"
    system "clear" or system "cls"
    english(english_list)
  when "2"
    system "clear" or system "cls"
    modification_menu
  when "3"
    system "clear" or system "cls"
    tutorial
  when "4"
    exit
  else
    main_menu
  end
end

def english(received_list)
  puts "Drag in here the document you want to analyze:"
  provided_path = gets.chomp.to_s.tr("'", "").rstrip
  execution(provided_path, received_list)
end

def execution (provided_path, forbidden_list)
  data = File.read provided_path
  doc = Yomu.read :text, data

  text_listed = doc.to_s.downcase.gsub(/\p{^Alnum}/, " ").split(" ")
  pruned_text = text_listed - forbidden_list

  frequency = Hash.new 0
  pruned_text.each { |word| frequency[word] += 1 }
  sorted_frequency = Hash[frequency.sort_by{ | word, times | -times }[0..20]]
  toptimes = sorted_frequency.values
  topwords = sorted_frequency.keys

  puts Hirb::Helpers::AutoTable.render(sorted_frequency, headers: {0 => 'Word', 1 => 'Times'}, description: false)
  puts "-----------------------------------------------------"
  main_menu

end

In the "main_menu" the user can choose to analyze the document. If so, the script asks the path of the document in the function "user_doc_path", which takes also the list of forbidden words in English from a .txt file which comes in the program by default. It is a simple file named "english_list.txt" which contains all the words to be removed from the document separated by a line break. So, the user provides the path of the document and the script opens it using 'yum' gem and sort it to get the words in an array. After that, the script creates a hash that stores the number of times each word is repeated throughout the text. Then, the program orders the hash, reverses it and get the 15 top results which finally output in a table provided by 'hirb' gem.

If the user choose to modify the list of words that are removed from the document, he or she is redirected to the "forbidden_list_eng_modify.rb":

def remove_words(list_to_remove)
  puts "Write the words you want to remove from the list. If there are more than one, divide them by commas."
  remove_propose = gets.chomp.tr(" ", "").to_s.split(",")
  new_list = list_to_remove - remove_propose

  File.new("languages/english/english_backup.txt", "w")
  File.open("languages/english/english_backup.txt", "w+") do |f|
  f.puts(list_to_remove)
  end

  File.open("languages/english/english_list.txt", "w+") do |f|
  f.puts(new_list)
  end

  puts "The words have been removed. Do you want to see the new list? [Yes/No]"
  showit = gets.chomp.tr(" ", "").to_s
  case showit
  when "Yes", "yes", "Y", "y"
    system "clear" or system "cls"
    print new_list
    puts "-----------------------------------------------------"
    modification_menu
  when "No", "no", "N", "n"
    system "clear" or system "cls"
    modification_menu
  end
end

def add_words(list_to_add)
  puts "Write the words you want to add to the list. If there are more than one, divide them by commas."
  add_propose = gets.chomp.tr(" ", "").to_s.split(",")
  new_list = list_to_add + add_propose

  File.new("languages/english/english_backup.txt", "w")
  File.open("languages/english/english_backup.txt", "w+") do |f|
  f.puts(list_to_add)
  end
  File.open("languages/english/english_list.txt", "w+") do |f|
  f.puts(new_list)
  end

  puts "The words have been added. Do you want to see the new list? [Yes/No]"
  showit = gets.chomp.tr(" ", "").to_s
  case showit
  when "Yes", "yes", "Y", "y"
    system "clear" or system "cls"
    print new_list
    puts "-----------------------------------------------------"
    modification_menu
  when "No", "no", "N", "n"
    puts "-----------------------------------------------------"
    modification_menu
  end
end

def modification_menu
retrieve_forbidden_words = File.read('languages/english/english_list.txt')
original_list = retrieve_forbidden_words.split("\n")

puts "This list contains all the words that are automatically removed from the document you want to analyze. The words in it are widely-used words with no special meaning like connectors or prepositions. However, you can modify the list as you want."
puts "1: Show all the words of the list."
puts "2: Remove words."
puts "3: Add words."
puts "4: Save the new list for other sessions."
puts "5: Return to the main menu."
list_choice = gets.chomp.to_s
case list_choice
when "1"
  system "clear" or system "cls"
  print original_list
  puts ""
  puts "-----------------------------------------------------"
  modification_menu
when "2"
  system "clear" or system "cls"
  remove_words(original_list)
when "3"
  system "clear" or system "cls"
  add_words(original_list)
when "4"
  backup_existence = File.file?('languages/english/english_backup.txt')
  if backup_existence == TRUE
    File.delete('languages/english/english_backup.txt')
    system "clear" or system "cls"
    puts "Changes has been succesfully saved."
    puts "-----------------------------------------------------"
    modification_menu
  end
  if backup_existence == FALSE
    system "clear" or system "cls"
    puts "The list hasn't been modified, so it's nothing to save."
    puts "-----------------------------------------------------"
    modification_menu
  end
when "5"
  system "clear" or system "cls"
  main_menu
else
  system "clear" or system "cls"
  modification_menu
end
end

This script shows the user a menu which offers the possibility of showing the list, adding or removing some words and saving the changes for ever. The first option, show the complete list, just prints the output of the "english_list.txt", which is the file where the words are stored. The "add_words", as the "remove_words" function, asks the user to write the words that want to get removed separating them by commas. The script just removes the provided array from the default list. At this point, the script creates a "english_backup.txt" which is the back up in case the user closes the program without saving. The modified list is named "english_list.txt", and it works as the new default list, so the user can analyze more texts in the same session with that list. If the user chooses to save the changes, the "english_backup.txt" is destroyed and the new "english_list.txt" is the main list. If the user exits, the "english_backup.txt" will get its former name ("english_list.txt") and the unsaved list file will be removed. This is achieved in the former script, "wordsworth.rb", when END is detected it proceeds to remove the unsaved file and restore the backup as main list.

Answer 1

This is a really interesting project, and looks pretty solid. I'm just going to go through and give you some ruby style tips as I see them. I've annotated all of the files with my thoughts.

First of all, you use lists in a couple places. Because of that, I'd suggest that you create a generic list method right above your #language method:

def list(*options, retry_msg: '')
  options = Array(options).flatten.map(&:to_s).uniq
  invalid_opts = (0..options.length).map(&:to_s)
  throw "You cannot pass a number as an option" if options.any? { |opt| invalid_opts.include?(opt) }
  loop do
    puts options.map.with_index { |opt, i| "#{i}. #{opt}" }
    selection = gets.chomp
    system "clear" or system "cls"
    if (0..options.length-1).map(&:to_s).include? selection
      return options[selection.to_i]
    else
      selection = options.select { |opt| opt.downcase == selection.downcase }.first
      return selection unless selection.nil?
    end
    puts retry_msg
  end
end

Side note: Since this would remove your lists throughout, I just wanted to add some code style tips in case you decided to leave the list in. Here are my comments on the list in your #language method:

def language
  # I would simplify the listing of languages (and make it more extensible)
  # by having an array of languages you support.
  puts "Choose your language:"
  # I've created the array using the %w literal, which creates an array
  # of words split on the spaces.
  languages = %w[Català English Castellano]
  # puts "1. Català"
  # puts "2. English"
  # puts "3. Castellano"
  # And then print them out: (puts automatically adds newlines
  # between the array elements)
  puts languages.map.with_index { |lang, index| "#{index}. #{lang}" } 
  idiom = gets.chomp.to_s
  # Here, you can convert the idiom to lowercase so you don't need
  # to have different capitalizations in your when statements
  case idiom.downcase
  when "1", "català"#, "Català", "CATALÀ"
    system "clear" or system "cls"
    menu_principal_cat
  when "2", "english"#, "English", "ENGLISH"
    system "clear" or system "cls"
    main_menu
  when "3", "castellano"#, "Castellano", "CASTELLANO"
    system "clear" or system "cls"
    menu_principal_cast
  else
    system "clear" or system "cls"
    puts "You have to choose a language in the list."
    puts '-'*53 # You can use multiplication instead of typing the char so many times
    language
  end
end

Anyways, here's an actual review of your main file:

require 'hirb'
require 'yomu'
require 'fileutils'

require_relative "languages/english/forbidden_list_modify_eng.rb"
require_relative "languages/english/core.rb"

# The addition of the list method means that all this code doesn't need
# to be wrapped in a language method
# def language
  puts "Choose your language:"
  # puts "1. Català"
  # puts "2. English"
  # puts "3. Castellano"
  # idiom = gets.chomp.to_s
  retry_msg  = "You have to choose a language in the list.\n"
  retry_msg += '-'*53
  case list(%w[Català English Castellano], retry_msg: retry_msg)
  when "Català"
    menu_principal_cat
  when "English"
    main_menu
  when "Castellano"
    menu_principal_cast
  end
# end

# language()

And, for the second file:

def tutorial
  # If this is really long, it might be a good idea to put
  # the contents of the tutorial in a text file and instead
  # use puts File.read("path/to/file")
  puts "The program works like that... (blah blah blah)"
end

def main_menu
  # I've renamed the variable to be forbidden_words (from retrieve_forbidden_words) because the variable is the forbidden words, not the action of retrieving them.
  forbidden_words = File.read('languages/english/english_list.txt')
  english_list = forbidden_words.split("\n")
  # Here we can use the magic list method
  puts "Choose what you want to do by writing its number:"
  # puts "1: Analyze a document."
  # puts "2: Modify the list of forbidden words."
  # puts "3: Help me to use this program."
  # puts "4: Exit the program."
  # main_choice = gets.chomp.to_s
  # case main_choice
  options = []
  options << 'Analyze a document.'
  options << 'Modify the list of forbidden words.'
  options << 'Help me to use this program.'
  options << 'Exit the program.'
  case list(options)
  when 'Analyze a document.'
    english(english_list)
  when 'Modify the list of forbidden words.'
    modification_menu
  when 'Help me to use this program.'
    tutorial
  when 'Exit the program.'
    exit
  end
end

def english(received_list)
  puts "Drag in here the document you want to analyze:"
  provided_path = gets.chomp.to_s.tr("'", "").rstrip
  execution(provided_path, received_list)
end

def execution (provided_path, forbidden_list)
  data = File.read provided_path
  doc = Yomu.read :text, data

  text_listed = doc.to_s.downcase.gsub(/\p{^Alnum}/, " ").split(" ")
  pruned_text = text_listed - forbidden_list

  frequency = Hash.new 0
  pruned_text.each { |word| frequency[word] += 1 }
  sorted_frequency = Hash[frequency.sort_by{ | word, times | -times }[0..20]]
  toptimes = sorted_frequency.values
  topwords = sorted_frequency.keys

  puts Hirb::Helpers::AutoTable.render(sorted_frequency, headers: {0 => 'Word', 1 => 'Times'}, description: false)
  puts "-----------------------------------------------------"
  main_menu

end

And, the final file:

def remove_words(list_to_remove)
  puts "Write the words you want to remove from the list. If there are more than one, divide them by commas."
  remove_propose = gets.chomp.tr(" ", "").to_s.split(",")
  new_list = list_to_remove - remove_propose

  File.new("languages/english/english_backup.txt", "w")
  File.open("languages/english/english_backup.txt", "w+") do |f|
  f.puts(list_to_remove)
  end

  File.open("languages/english/english_list.txt", "w+") do |f|
  f.puts(new_list)
  end

  puts "The words have been removed. Do you want to see the new list? [Yes/No]"
  showit = gets.chomp.tr(" ", "").to_s
  case showit
  when "Yes", "yes", "Y", "y"
    system "clear" or system "cls"
    print new_list
    puts "-----------------------------------------------------"
    modification_menu
  when "No", "no", "N", "n"
    system "clear" or system "cls"
    modification_menu
  end
end

def add_words(list_to_add)
  puts "Write the words you want to add to the list. If there are more than one, divide them by commas."
  add_propose = gets.chomp.tr(" ", "").to_s.split(",")
  new_list = list_to_add + add_propose

  File.new("languages/english/english_backup.txt", "w")
  File.open("languages/english/english_backup.txt", "w+") do |f|
  f.puts(list_to_add)
  end
  File.open("languages/english/english_list.txt", "w+") do |f|
  f.puts(new_list)
  end

  puts "The words have been added. Do you want to see the new list? [Yes/No]"
  showit = gets.chomp.tr(" ", "").to_s
  case showit
  when "Yes", "yes", "Y", "y"
    system "clear" or system "cls"
    print new_list
    puts "-----------------------------------------------------"
    modification_menu
  when "No", "no", "N", "n"
    puts "-----------------------------------------------------"
    modification_menu
  end
end

def modification_menu
retrieve_forbidden_words = File.read('languages/english/english_list.txt')
original_list = retrieve_forbidden_words.split("\n")

puts "This list contains all the words that are automatically removed from the document you want to analyze. The words in it are widely-used words with no special meaning like connectors or prepositions. However, you can modify the list as you want."
puts "1: Show all the words of the list."
puts "2: Remove words."
puts "3: Add words."
puts "4: Save the new list for other sessions."
puts "5: Return to the main menu."
list_choice = gets.chomp.to_s
case list_choice
when "1"
  system "clear" or system "cls"
  print original_list
  puts ""
  puts "-----------------------------------------------------"
  modification_menu
when "2"
  system "clear" or system "cls"
  remove_words(original_list)
when "3"
  system "clear" or system "cls"
  add_words(original_list)
when "4"
  backup_existence = File.file?('languages/english/english_backup.txt')
  if backup_existence == TRUE
    File.delete('languages/english/english_backup.txt')
    system "clear" or system "cls"
    puts "Changes has been succesfully saved."
    puts "-----------------------------------------------------"
    modification_menu
  end
  if backup_existence == FALSE
    system "clear" or system "cls"
    puts "The list hasn't been modified, so it's nothing to save."
    puts "-----------------------------------------------------"
    modification_menu
  end
when "5"
  system "clear" or system "cls"
  main_menu
else
  system "clear" or system "cls"
  modification_menu
end
end

---

Side note: My apologies for taking so long to get back to this, I just got busy IRL and couldn't finish.